[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:49:08 dist-test-slave-q25r krb5kdc[21020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:49:08 dist-test-slave-q25r krb5kdc[21020](info): set up 2 sockets
Sep 27 13:49:08 dist-test-slave-q25r krb5kdc[21020](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 13:49:09.137149 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.006s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:49:09 dist-test-slave-q25r krb5kdc[21020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:49:09.330845 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:35509
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:35509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:09.704262 21033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:09.704706 21033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:09.704838 21033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:09.713289 21033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:09.713429 21033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:09.713483 21033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:09.713572 21033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:09.713639 21033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:09.724771 21033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:35509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:35509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:09.726485 21033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:09.728058 21033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:09.742502 21039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:09.742686 21033 server_base.cc:1034] running on GCE node
W20260927 13:49:09.742579 21038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:09.743083 21041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:09.744098 21033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:09.745887 21033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:09.747114 21033 hybrid_clock.cc:648] HybridClock initialized: now 1790516949747032 us; error 68 us; skew 500 ppm
I20260927 13:49:09.747502 21033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:09 dist-test-slave-q25r krb5kdc[21020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516949, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:09.755642 21033 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:49:09.757510 21033 webserver.cc:460] Webserver started at http://127.20.133.126:35407/ using document root <none> and password file <none>
I20260927 13:49:09.758620 21033 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:09.758797 21033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:09.759150 21033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:09.762584 21033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "86774dc2540e4a5ab976295858272a96"
format_stamp: "Formatted at 2026-09-27 13:49:09 on dist-test-slave-q25r"
server_key: "abecb49019e30c1bf28fd2611908d050"
server_key_iv: "26ef9fc6bfc7af864ea5c99d02734d84"
server_key_version: "encryptionkey@0"
I20260927 13:49:09.763417 21033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "86774dc2540e4a5ab976295858272a96"
format_stamp: "Formatted at 2026-09-27 13:49:09 on dist-test-slave-q25r"
server_key: "abecb49019e30c1bf28fd2611908d050"
server_key_iv: "26ef9fc6bfc7af864ea5c99d02734d84"
server_key_version: "encryptionkey@0"
I20260927 13:49:09.769775 21033 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 13:49:09.773873 21048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:09.775465 21033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:49:09.775662 21033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "86774dc2540e4a5ab976295858272a96"
format_stamp: "Formatted at 2026-09-27 13:49:09 on dist-test-slave-q25r"
server_key: "abecb49019e30c1bf28fd2611908d050"
server_key_iv: "26ef9fc6bfc7af864ea5c99d02734d84"
server_key_version: "encryptionkey@0"
I20260927 13:49:09.775856 21033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:09.799767 21033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:09.804320 21033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:09.804620 21033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:09.824456 21099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:35509 every 8 connection(s)
I20260927 13:49:09.824450 21033 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:35509
I20260927 13:49:09.826148 21033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:09.831216 21100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:09.833189 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 21033
I20260927 13:49:09.833571 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:09.836376 21013 external_mini_cluster.cc:1428] Setting key 81c69eba33c92631d8a5f84b3322fa7a
I20260927 13:49:09.847268 21100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96: Bootstrap starting.
I20260927 13:49:09.852205 21100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:09.853899 21100 log.cc:824] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:09.858130 21100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96: No bootstrap required, opened a new log
I20260927 13:49:09.868999 21100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774dc2540e4a5ab976295858272a96" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35509 } }
I20260927 13:49:09.869592 21100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:09.869733 21100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86774dc2540e4a5ab976295858272a96, State: Initialized, Role: FOLLOWER
I20260927 13:49:09.870612 21100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774dc2540e4a5ab976295858272a96" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35509 } }
I20260927 13:49:09.870955 21100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:09.871114 21100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:09.871317 21100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:09.875092 21100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774dc2540e4a5ab976295858272a96" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35509 } }
Sep 27 13:49:09 dist-test-slave-q25r krb5kdc[21020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:09.875840 21100 leader_election.cc:302] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86774dc2540e4a5ab976295858272a96; no voters:
I20260927 13:49:09.876605 21100 leader_election.cc:288] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:09.876741 21105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:09.878166 21105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [term 1 LEADER]: Becoming Leader. State: Replica: 86774dc2540e4a5ab976295858272a96, State: Running, Role: LEADER
I20260927 13:49:09.878916 21105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774dc2540e4a5ab976295858272a96" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35509 } }
I20260927 13:49:09.879657 21100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:09.882426 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86774dc2540e4a5ab976295858272a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774dc2540e4a5ab976295858272a96" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35509 } } }
I20260927 13:49:09.882879 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:09.886821 21107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86774dc2540e4a5ab976295858272a96. Latest consensus state: current_term: 1 leader_uuid: "86774dc2540e4a5ab976295858272a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774dc2540e4a5ab976295858272a96" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35509 } } }
I20260927 13:49:09.887239 21107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:09.890560 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:09.842085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.20.133.126:35509)
0927 13:49:09.843307 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:09.843326 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:09.845308 (+ 1982us) server_negotiation.cc:407] Connection header received
0927 13:49:09.858080 (+ 12772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:09.858116 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:09.858651 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:09.859061 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:09.863806 (+ 4745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:09.865417 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:09.868265 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:09.869165 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:09.869346 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:09.877792 (+ 8446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:09.877805 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:09.877827 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:09.877896 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:09.882059 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:09.883286 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:09.883294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:09.883300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:09.883546 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:09.884149 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:09.884154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:09.884157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:09.884983 (+ 826us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:09.885312 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:09.888354 (+ 3042us) server_negotiation.cc:299] Negotiation successful
0927 13:49:09.888857 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":510,"threads_started":1}
I20260927 13:49:09.892298 21110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:09.899978 21110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:09.912634 21110 catalog_manager.cc:1347] Generated new cluster ID: 7c25e20f385b40729f9d75319a3fdc21
I20260927 13:49:09.912851 21110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:09.942687 21110 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:09.944808 21110 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:09.953258 21110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86774dc2540e4a5ab976295858272a96: Generated new TSK 0
I20260927 13:49:09.954443 21110 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:49:10.008198 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:35509
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:10.391055 21126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:10.391431 21126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:10.391569 21126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:10.399791 21126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:10.399941 21126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:10.400095 21126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:49:10.411242 21126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:35509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:10.413034 21126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:10.414690 21126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:10.430060 21132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:10.430102 21131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:10.430122 21134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:10.433936 21126 server_base.cc:1034] running on GCE node
I20260927 13:49:10.434648 21126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:10.435793 21126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:10.437036 21126 hybrid_clock.cc:648] HybridClock initialized: now 1790516950436968 us; error 62 us; skew 500 ppm
I20260927 13:49:10.437398 21126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:10 dist-test-slave-q25r krb5kdc[21020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516950, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:10.440675 21126 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:49:10.442399 21126 webserver.cc:460] Webserver started at http://127.20.133.65:39253/ using document root <none> and password file <none>
I20260927 13:49:10.443197 21126 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:10.443311 21126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:10.443621 21126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:10.446750 21126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "997d675b0d3a458ebcad2741cfce2090"
format_stamp: "Formatted at 2026-09-27 13:49:10 on dist-test-slave-q25r"
server_key: "5a6ad082165b5743105fa679a75e8559"
server_key_iv: "a9eccffecd8a37f1af98056a4933bfdf"
server_key_version: "encryptionkey@0"
I20260927 13:49:10.447568 21126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "997d675b0d3a458ebcad2741cfce2090"
format_stamp: "Formatted at 2026-09-27 13:49:10 on dist-test-slave-q25r"
server_key: "5a6ad082165b5743105fa679a75e8559"
server_key_iv: "a9eccffecd8a37f1af98056a4933bfdf"
server_key_version: "encryptionkey@0"
I20260927 13:49:10.453944 21126 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 13:49:10.458648 21141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:10.460742 21126 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 13:49:10.460964 21126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "997d675b0d3a458ebcad2741cfce2090"
format_stamp: "Formatted at 2026-09-27 13:49:10 on dist-test-slave-q25r"
server_key: "5a6ad082165b5743105fa679a75e8559"
server_key_iv: "a9eccffecd8a37f1af98056a4933bfdf"
server_key_version: "encryptionkey@0"
I20260927 13:49:10.461237 21126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:10.480110 21126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:10.485426 21126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:10.485767 21126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:10.487025 21126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:10.488761 21126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:10.488854 21126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:10.488987 21126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:10.489053 21126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:10.526508 21126 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:37211
I20260927 13:49:10.526569 21253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:37211 every 8 connection(s)
I20260927 13:49:10.528092 21126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:49:10.530228 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 21126
I20260927 13:49:10.530455 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:49:10.531206 21013 external_mini_cluster.cc:1428] Setting key 7040faa83c717d693a758c538d74af73
Sep 27 13:49:10 dist-test-slave-q25r krb5kdc[21020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516950, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
I20260927 13:49:10.551705 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:10.531058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:43315 (local address 127.20.133.126:35509)
0927 13:49:10.531668 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:10.531676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:10.532928 (+ 1252us) server_negotiation.cc:407] Connection header received
0927 13:49:10.534889 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:10.534895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:10.534995 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:10.535152 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:10.536970 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:10.538056 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:10.539841 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:10.540292 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:10.540443 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:10.544977 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:10.544984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:10.544987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:10.545039 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:10.547933 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:10.549210 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:10.549215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:10.549217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:10.549355 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:10.549902 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:10.549906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:10.549908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:10.550219 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:10.550361 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:10.551162 (+ 801us) server_negotiation.cc:299] Negotiation successful
0927 13:49:10.551379 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":271,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
I20260927 13:49:10.552691 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:10.531571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:35509 (local address 127.20.133.65:43315)
0927 13:49:10.532736 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:10.532794 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:10.534578 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:10.535368 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:10.535378 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:10.535940 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:10.536777 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:10.536799 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:10.538270 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:10.538275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:10.539638 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:10.539653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:10.540551 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:10.540559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:10.540888 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:10.541680 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:10.541711 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:10.544675 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:10.548169 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:10.548183 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:10.548205 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:10.549028 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:10.549479 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:10.549484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:10.549488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:10.549779 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:10.550366 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:10.550374 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:10.550771 (+ 397us) client_negotiation.cc:769] Sending connection context
0927 13:49:10.551012 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 13:49:10.551458 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":343,"threads_started":1}
I20260927 13:49:10.554879 21254 heartbeater.cc:347] Connected to a master server at 127.20.133.126:35509
I20260927 13:49:10.555217 21254 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:10.556000 21254 heartbeater.cc:511] Master 127.20.133.126:35509 requested a full tablet report, sending...
I20260927 13:49:10.558717 21065 ts_manager.cc:194] Registered new tserver with Master: 997d675b0d3a458ebcad2741cfce2090 (127.20.133.65:37211)
I20260927 13:49:10.560464 21065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:43315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:49:10.576668 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:35509
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:10.915959 21260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:10.916399 21260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:10.916555 21260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:10.924695 21260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:10.924811 21260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:10.924962 21260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:49:10.934899 21260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:35509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:10.936264 21260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:10.937668 21260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:10.950958 21265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:10.950995 21266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:10.951020 21268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:10.951653 21260 server_base.cc:1034] running on GCE node
I20260927 13:49:10.952396 21260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:10.953436 21260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:10.954676 21260 hybrid_clock.cc:648] HybridClock initialized: now 1790516950954618 us; error 52 us; skew 500 ppm
I20260927 13:49:10.955015 21260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:10 dist-test-slave-q25r krb5kdc[21020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516950, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:10.958235 21260 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:49:10.959753 21260 webserver.cc:460] Webserver started at http://127.20.133.66:40219/ using document root <none> and password file <none>
I20260927 13:49:10.960474 21260 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:10.960606 21260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:10.960913 21260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:10.963871 21260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "3f90c00a019444d7844d2d4be6d4e198"
format_stamp: "Formatted at 2026-09-27 13:49:10 on dist-test-slave-q25r"
server_key: "ecb9d43e8f43a057f398da01f7283442"
server_key_iv: "262922b53975c28090c64f9026dca54d"
server_key_version: "encryptionkey@0"
I20260927 13:49:10.964722 21260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "3f90c00a019444d7844d2d4be6d4e198"
format_stamp: "Formatted at 2026-09-27 13:49:10 on dist-test-slave-q25r"
server_key: "ecb9d43e8f43a057f398da01f7283442"
server_key_iv: "262922b53975c28090c64f9026dca54d"
server_key_version: "encryptionkey@0"
I20260927 13:49:10.970136 21260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:49:10.973817 21275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:10.975258 21260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:49:10.975425 21260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "3f90c00a019444d7844d2d4be6d4e198"
format_stamp: "Formatted at 2026-09-27 13:49:10 on dist-test-slave-q25r"
server_key: "ecb9d43e8f43a057f398da01f7283442"
server_key_iv: "262922b53975c28090c64f9026dca54d"
server_key_version: "encryptionkey@0"
I20260927 13:49:10.975613 21260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:10.993779 21260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:10.998399 21260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:10.998677 21260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:10.999655 21260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:11.001108 21260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:11.001185 21260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:11.001304 21260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:11.001358 21260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:11.032783 21260 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:39573
I20260927 13:49:11.032799 21387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:39573 every 8 connection(s)
I20260927 13:49:11.034094 21260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:49:11.036274 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 21260
I20260927 13:49:11.036485 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:49:11.036965 21013 external_mini_cluster.cc:1428] Setting key c693fe14a5698a7dd9b2f02bdd021e68
Sep 27 13:49:11 dist-test-slave-q25r krb5kdc[21020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516950, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
I20260927 13:49:11.055881 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:11.036603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:46173 (local address 127.20.133.126:35509)
0927 13:49:11.036832 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:11.036838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:11.038011 (+ 1173us) server_negotiation.cc:407] Connection header received
0927 13:49:11.040065 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:11.040069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:11.040136 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:11.040207 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:11.042218 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.042967 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.044876 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.045467 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.045591 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:11.049421 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:11.049428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:11.049433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:11.049479 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:11.052799 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.053593 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.053599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.053603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.053757 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.054230 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.054234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.054236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.054540 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:11.054662 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:11.055358 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 13:49:11.055585 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
I20260927 13:49:11.056869 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:11.037120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:35509 (local address 127.20.133.66:46173)
0927 13:49:11.037859 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:11.037901 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:11.039770 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:11.040496 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:11.040508 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:11.041230 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:11.042022 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:11.042049 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.043118 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.043123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:11.044728 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:11.044742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.045702 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.045707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:11.045936 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:11.046618 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:11.046641 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:11.049133 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:11.052953 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:11.052961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:11.052978 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:11.053453 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:11.053877 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:11.053882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:11.053886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:11.054115 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:11.054667 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:11.054674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:11.055031 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 13:49:11.055268 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 13:49:11.055672 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":283,"threads_started":1}
I20260927 13:49:11.058300 21388 heartbeater.cc:347] Connected to a master server at 127.20.133.126:35509
I20260927 13:49:11.058655 21388 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:11.059514 21388 heartbeater.cc:511] Master 127.20.133.126:35509 requested a full tablet report, sending...
I20260927 13:49:11.061228 21065 ts_manager.cc:194] Registered new tserver with Master: 3f90c00a019444d7844d2d4be6d4e198 (127.20.133.66:39573)
I20260927 13:49:11.062238 21065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:46173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:49:11.077296 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:35509
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:11.403638 21393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:11.403961 21393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:11.404088 21393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:11.411643 21393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:11.411798 21393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:11.411950 21393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:49:11.422078 21393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:35509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:11.423463 21393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:11.424808 21393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:11.437817 21398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:11.437887 21401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:11.437912 21399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:11.438885 21393 server_base.cc:1034] running on GCE node
I20260927 13:49:11.439515 21393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:11.440527 21393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:11.441753 21393 hybrid_clock.cc:648] HybridClock initialized: now 1790516951441693 us; error 51 us; skew 500 ppm
I20260927 13:49:11.442085 21393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:11 dist-test-slave-q25r krb5kdc[21020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516951, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:11.445276 21393 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:49:11.446832 21393 webserver.cc:460] Webserver started at http://127.20.133.67:46237/ using document root <none> and password file <none>
I20260927 13:49:11.447525 21393 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:11.447643 21393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:11.447953 21393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:11.451097 21393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "7540c086a74948deb12c23aff732572e"
format_stamp: "Formatted at 2026-09-27 13:49:11 on dist-test-slave-q25r"
server_key: "7d95a8d8688b1dccc8a1b89cb4c314d8"
server_key_iv: "315e8a4e6e740c627807a4e22e8591a6"
server_key_version: "encryptionkey@0"
I20260927 13:49:11.451996 21393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "7540c086a74948deb12c23aff732572e"
format_stamp: "Formatted at 2026-09-27 13:49:11 on dist-test-slave-q25r"
server_key: "7d95a8d8688b1dccc8a1b89cb4c314d8"
server_key_iv: "315e8a4e6e740c627807a4e22e8591a6"
server_key_version: "encryptionkey@0"
I20260927 13:49:11.457563 21393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:49:11.461230 21408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:11.463104 21393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:49:11.463315 21393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "7540c086a74948deb12c23aff732572e"
format_stamp: "Formatted at 2026-09-27 13:49:11 on dist-test-slave-q25r"
server_key: "7d95a8d8688b1dccc8a1b89cb4c314d8"
server_key_iv: "315e8a4e6e740c627807a4e22e8591a6"
server_key_version: "encryptionkey@0"
I20260927 13:49:11.463553 21393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:11.494867 21393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:11.499142 21393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:11.499425 21393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:11.500422 21393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:11.501770 21393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:11.501847 21393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:11.501969 21393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:11.502029 21393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:11.533097 21393 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:37337
I20260927 13:49:11.533139 21520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:37337 every 8 connection(s)
I20260927 13:49:11.534314 21393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:49:11.534824 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 21393
I20260927 13:49:11.535003 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:49:11.535539 21013 external_mini_cluster.cc:1428] Setting key 57bf82f242a137e6e28b92b69ee93ef2
Sep 27 13:49:11 dist-test-slave-q25r krb5kdc[21020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516951, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:11.555135 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:11.536868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:37949 (local address 127.20.133.126:35509)
0927 13:49:11.537044 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:11.537050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:11.538421 (+ 1371us) server_negotiation.cc:407] Connection header received
0927 13:49:11.540465 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:11.540470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:11.540538 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:11.540617 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:11.542107 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.542878 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.544561 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.545003 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.545110 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:11.548780 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:11.548787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:11.548791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:11.548830 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:11.551835 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.552736 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.552740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.552743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.552892 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.553341 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.553346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.553348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.553639 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:11.553793 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:11.554537 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 13:49:11.554782 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:49:11.556181 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:11.537448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:35509 (local address 127.20.133.67:37949)
0927 13:49:11.538267 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:11.538314 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:11.540154 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:11.540828 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:11.540837 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:11.541342 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:11.541936 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:11.541955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.543030 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.543035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:11.544389 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:11.544406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.545225 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.545231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:11.545464 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:11.546140 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:11.546163 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:11.548453 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:11.552018 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:11.552027 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:11.552043 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:11.552601 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:11.553003 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:11.553007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:11.553009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:11.553219 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:11.553774 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:11.553783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:11.554157 (+ 374us) client_negotiation.cc:769] Sending connection context
0927 13:49:11.554417 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 13:49:11.555000 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":345,"threads_started":1}
I20260927 13:49:11.557742 21521 heartbeater.cc:347] Connected to a master server at 127.20.133.126:35509
I20260927 13:49:11.558075 21521 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:11.558805 21521 heartbeater.cc:511] Master 127.20.133.126:35509 requested a full tablet report, sending...
I20260927 13:49:11.560266 21065 ts_manager.cc:194] Registered new tserver with Master: 7540c086a74948deb12c23aff732572e (127.20.133.67:37337)
I20260927 13:49:11.561154 21065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:37949
I20260927 13:49:11.564167 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:49:11.564532 21254 heartbeater.cc:503] Master 127.20.133.126:35509 was elected leader, sending a full tablet report...
Sep 27 13:49:11 dist-test-slave-q25r krb5kdc[21020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516951, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:49:11 dist-test-slave-q25r krb5kdc[21020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516951, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:11.614677 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:11.595643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.20.133.65:37211)
0927 13:49:11.596562 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:11.596570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:11.596606 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:49:11.596672 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:11.596679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:11.596886 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:11.597014 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:11.598625 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.599710 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.601512 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.602176 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.602303 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:11.607985 (+ 5682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:11.608006 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:11.608024 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:11.608075 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:11.611100 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.611989 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.611997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.612000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.612153 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.612699 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.612706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.612708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.613305 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:11.613641 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:11.614101 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 13:49:11.614373 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":623,"threads_started":1}
W20260927 13:49:11.616034 21169 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33070
Sep 27 13:49:11 dist-test-slave-q25r krb5kdc[21020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:49:11 dist-test-slave-q25r krb5kdc[21020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:11.661491 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:11.646235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.20.133.65:37211)
0927 13:49:11.646523 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:11.646529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:11.646768 (+ 239us) server_negotiation.cc:407] Connection header received
0927 13:49:11.646960 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:11.646964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:11.647030 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:11.647115 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:11.648509 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.649226 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.650431 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.650883 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.651001 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:11.655097 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:11.655104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:11.655107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:11.655148 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:11.658425 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.659299 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.659305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.659307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.659473 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.659898 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.659903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.659905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.660215 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:11.660504 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:11.660876 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 13:49:11.661126 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 13:49:11.663362 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 21126
I20260927 13:49:11.677023 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:11.664547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.20.133.65:37211)
0927 13:49:11.664759 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:11.664765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:11.664930 (+ 165us) server_negotiation.cc:407] Connection header received
0927 13:49:11.665124 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:11.665130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:11.665212 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:11.665298 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:11.666575 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.667334 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.668769 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.669198 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.669315 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:11.671646 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:11.671651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:11.671655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:11.671694 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:11.674378 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.675115 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.675121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.675123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.675276 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.675563 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.675566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.675567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.675830 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:11.675997 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:11.676412 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 13:49:11.676623 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:49:11.677963 21249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:11.765331 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 21126
I20260927 13:49:11.778534 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 21260
Sep 27 13:49:11 dist-test-slave-q25r krb5kdc[21020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:49:11.795321 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:11.779528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.20.133.66:39573)
0927 13:49:11.780049 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:11.780057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:11.780083 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:49:11.780150 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:11.780156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:11.780462 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:11.780672 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:11.781983 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.783055 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.784506 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.785156 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.785285 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:11.789497 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:11.789512 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:11.789529 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:11.789571 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:11.792457 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.793238 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.793245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.793248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.793374 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.793804 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.793809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.793812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.794197 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:11.794424 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:11.794725 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 13:49:11.794944 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
I20260927 13:49:11.796741 21383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:11.885458 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 21260
I20260927 13:49:11.898701 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 21393
Sep 27 13:49:11 dist-test-slave-q25r krb5kdc[21020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:49:11.917979 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:11.900060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.20.133.67:37337)
0927 13:49:11.900907 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:11.900916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:11.900954 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:49:11.901038 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:11.901047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:11.901333 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:11.901531 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:11.902976 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.904175 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.905736 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:11.906376 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:11.906517 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:11.911278 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:11.911293 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:11.911312 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:11.911355 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:11.914615 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.915554 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.915564 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.915567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.915721 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:11.916105 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:11.916109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:11.916112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:11.916589 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:11.916970 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:11.917248 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 13:49:11.917580 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":458,"threads_started":1}
I20260927 13:49:11.919922 21516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:12.015421 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 21393
I20260927 13:49:12.028389 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 21033
I20260927 13:49:12.029969 21095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:12.147238 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 21033
2026-09-27T13:49:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4046 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:49:12 dist-test-slave-q25r krb5kdc[21542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:49:12 dist-test-slave-q25r krb5kdc[21542](info): set up 2 sockets
Sep 27 13:49:12 dist-test-slave-q25r krb5kdc[21542](info): commencing operation
krb5kdc: starting...
W20260927 13:49:13.164930 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:49:13 dist-test-slave-q25r krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:49:13.367770 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:34655
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:34655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:13.741915 21555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:13.742301 21555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:13.742401 21555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:13.750483 21555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:13.750648 21555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:13.750696 21555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:13.750759 21555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:13.750809 21555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:13.762065 21555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:34655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:34655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:13.763784 21555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:13.765384 21555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:13.779371 21561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:13.779312 21563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:13.779973 21560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:13.780812 21555 server_base.cc:1034] running on GCE node
I20260927 13:49:13.781816 21555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:13.783782 21555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:13.785027 21555 hybrid_clock.cc:648] HybridClock initialized: now 1790516953784970 us; error 63 us; skew 500 ppm
I20260927 13:49:13.785455 21555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:13 dist-test-slave-q25r krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516953, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:13.789101 21555 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:49:13.791090 21555 webserver.cc:460] Webserver started at http://127.20.133.126:45169/ using document root <none> and password file <none>
I20260927 13:49:13.791927 21555 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:13.792042 21555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:13.792404 21555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:13.795881 21555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "4bd530c72c3b48f3ade3d9a5d348243f"
format_stamp: "Formatted at 2026-09-27 13:49:13 on dist-test-slave-q25r"
server_key: "79aceb12a5154a463ae8f048b33fd221"
server_key_iv: "745b24044b35e029530a7ca5be7b53a3"
server_key_version: "encryptionkey@0"
I20260927 13:49:13.796720 21555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "4bd530c72c3b48f3ade3d9a5d348243f"
format_stamp: "Formatted at 2026-09-27 13:49:13 on dist-test-slave-q25r"
server_key: "79aceb12a5154a463ae8f048b33fd221"
server_key_iv: "745b24044b35e029530a7ca5be7b53a3"
server_key_version: "encryptionkey@0"
I20260927 13:49:13.802773 21555 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 13:49:13.807137 21570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:13.808964 21555 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:49:13.809177 21555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "4bd530c72c3b48f3ade3d9a5d348243f"
format_stamp: "Formatted at 2026-09-27 13:49:13 on dist-test-slave-q25r"
server_key: "79aceb12a5154a463ae8f048b33fd221"
server_key_iv: "745b24044b35e029530a7ca5be7b53a3"
server_key_version: "encryptionkey@0"
I20260927 13:49:13.809396 21555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:13.840149 21555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:13.845320 21555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:13.845634 21555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:13.865773 21555 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:34655
I20260927 13:49:13.865779 21621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:34655 every 8 connection(s)
I20260927 13:49:13.867340 21555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:13.869027 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 21555
I20260927 13:49:13.869282 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:13.870086 21013 external_mini_cluster.cc:1428] Setting key 5386c1388f3f606c10c2da629915f80b
I20260927 13:49:13.872190 21622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:49:13 dist-test-slave-q25r krb5kdc[21542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:13.885021 21622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f: Bootstrap starting.
I20260927 13:49:13.888500 21622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:13.889635 21622 log.cc:824] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:13.893196 21622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f: No bootstrap required, opened a new log
I20260927 13:49:13.893728 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:13.872267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.20.133.126:34655)
0927 13:49:13.873097 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:13.873110 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:13.873152 (+ 42us) server_negotiation.cc:407] Connection header received
0927 13:49:13.874876 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:13.874899 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:13.875336 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:13.875737 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:13.877428 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:13.879201 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:13.880659 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:13.881410 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:13.881625 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:13.885982 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:13.885991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:13.886009 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:13.886058 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:13.889147 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:13.889846 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:13.889853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:13.889860 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:13.890031 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:13.890488 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:13.890493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:13.890496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:13.891213 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:13.891436 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:13.892077 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 13:49:13.892525 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":339,"threads_started":1}
I20260927 13:49:13.901388 21622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd530c72c3b48f3ade3d9a5d348243f" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34655 } }
I20260927 13:49:13.901814 21622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:13.901911 21622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd530c72c3b48f3ade3d9a5d348243f, State: Initialized, Role: FOLLOWER
I20260927 13:49:13.902508 21622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd530c72c3b48f3ade3d9a5d348243f" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34655 } }
I20260927 13:49:13.902776 21622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:13.902926 21622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:13.903088 21622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:13.905433 21622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd530c72c3b48f3ade3d9a5d348243f" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34655 } }
I20260927 13:49:13.905954 21622 leader_election.cc:302] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd530c72c3b48f3ade3d9a5d348243f; no voters:
I20260927 13:49:13.906512 21622 leader_election.cc:288] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:13.906632 21627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:13.907637 21627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [term 1 LEADER]: Becoming Leader. State: Replica: 4bd530c72c3b48f3ade3d9a5d348243f, State: Running, Role: LEADER
I20260927 13:49:13.908161 21627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd530c72c3b48f3ade3d9a5d348243f" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34655 } }
I20260927 13:49:13.908684 21622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:13.910954 21628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bd530c72c3b48f3ade3d9a5d348243f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd530c72c3b48f3ade3d9a5d348243f" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34655 } } }
I20260927 13:49:13.911422 21628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:13.911100 21629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bd530c72c3b48f3ade3d9a5d348243f. Latest consensus state: current_term: 1 leader_uuid: "4bd530c72c3b48f3ade3d9a5d348243f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd530c72c3b48f3ade3d9a5d348243f" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34655 } } }
I20260927 13:49:13.911700 21629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:13.915032 21636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:13.920578 21636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:13.931324 21636 catalog_manager.cc:1347] Generated new cluster ID: 0a5f6efc13384bdba39b40c65403a834
I20260927 13:49:13.931459 21636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:13.961014 21636 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:13.962416 21636 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:13.976084 21636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bd530c72c3b48f3ade3d9a5d348243f: Generated new TSK 0
I20260927 13:49:13.977226 21636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:49:14.032089 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34655
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:14.391647 21648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:14.392014 21648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:14.392119 21648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:14.400185 21648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:14.400311 21648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:14.400477 21648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:49:14.411298 21648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:14.412719 21648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:14.414270 21648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:14.428546 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:14.428498 21656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:14.428470 21654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:14.429574 21648 server_base.cc:1034] running on GCE node
I20260927 13:49:14.430200 21648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:14.431202 21648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:14.432451 21648 hybrid_clock.cc:648] HybridClock initialized: now 1790516954432386 us; error 52 us; skew 500 ppm
I20260927 13:49:14.432803 21648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:14 dist-test-slave-q25r krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516954, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:14.436208 21648 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:49:14.437815 21648 webserver.cc:460] Webserver started at http://127.20.133.65:35941/ using document root <none> and password file <none>
I20260927 13:49:14.438545 21648 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:14.438671 21648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:14.438988 21648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:14.442103 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "f705b9cb70794465a2e985e5e6a2b999"
format_stamp: "Formatted at 2026-09-27 13:49:14 on dist-test-slave-q25r"
server_key: "9eb6468631e277aa504587c448413c9d"
server_key_iv: "1e10925f397eebf3c3dc99b67c5074c5"
server_key_version: "encryptionkey@0"
I20260927 13:49:14.442924 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "f705b9cb70794465a2e985e5e6a2b999"
format_stamp: "Formatted at 2026-09-27 13:49:14 on dist-test-slave-q25r"
server_key: "9eb6468631e277aa504587c448413c9d"
server_key_iv: "1e10925f397eebf3c3dc99b67c5074c5"
server_key_version: "encryptionkey@0"
I20260927 13:49:14.449044 21648 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 13:49:14.453629 21663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:14.455487 21648 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:49:14.455703 21648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "f705b9cb70794465a2e985e5e6a2b999"
format_stamp: "Formatted at 2026-09-27 13:49:14 on dist-test-slave-q25r"
server_key: "9eb6468631e277aa504587c448413c9d"
server_key_iv: "1e10925f397eebf3c3dc99b67c5074c5"
server_key_version: "encryptionkey@0"
I20260927 13:49:14.455924 21648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:14.480852 21648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:14.485144 21648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:14.485440 21648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:14.486492 21648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:14.487963 21648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:14.488044 21648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:14.488162 21648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:14.488214 21648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:14.521689 21648 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:38855
I20260927 13:49:14.521725 21775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:38855 every 8 connection(s)
I20260927 13:49:14.523057 21648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:49:14.531726 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 21648
I20260927 13:49:14.531955 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:49:14.532608 21013 external_mini_cluster.cc:1428] Setting key b49c6cac1bc85d807a6fadee626b16b7
Sep 27 13:49:14 dist-test-slave-q25r krb5kdc[21542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516954, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:14.544502 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:14.525682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:38173 (local address 127.20.133.126:34655)
0927 13:49:14.526096 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:14.526100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:14.527034 (+ 934us) server_negotiation.cc:407] Connection header received
0927 13:49:14.528987 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:14.528993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:14.529087 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:14.529193 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:14.530847 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:14.531754 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:14.533864 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:14.534337 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:14.534495 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:14.538691 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:14.538697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:14.538701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:14.538736 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:14.541231 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:14.542155 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:14.542159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:14.542162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:14.542286 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:14.542764 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:14.542767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:14.542768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:14.543027 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:14.543143 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:14.544002 (+ 859us) server_negotiation.cc:299] Negotiation successful
0927 13:49:14.544188 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":218,"threads_started":1}
I20260927 13:49:14.545902 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:14.526179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34655 (local address 127.20.133.65:38173)
0927 13:49:14.526882 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:14.526919 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:14.528683 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:14.529468 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:14.529477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:14.530024 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:14.530642 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:14.530661 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:14.532023 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:14.532035 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:14.533675 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:14.533693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:14.534568 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:14.534574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:14.534874 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:14.535639 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:14.535671 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:14.538419 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:14.541383 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:14.541392 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:14.541410 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:14.542013 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:14.542398 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:14.542402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:14.542405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:14.542669 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:14.543136 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:14.543144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:14.543587 (+ 443us) client_negotiation.cc:769] Sending connection context
0927 13:49:14.543901 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 13:49:14.544473 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":291,"threads_started":1}
I20260927 13:49:14.547540 21776 heartbeater.cc:347] Connected to a master server at 127.20.133.126:34655
I20260927 13:49:14.547865 21776 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:14.548650 21776 heartbeater.cc:511] Master 127.20.133.126:34655 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
I20260927 13:49:14.550822 21587 ts_manager.cc:194] Registered new tserver with Master: f705b9cb70794465a2e985e5e6a2b999 (127.20.133.65:38855)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
I20260927 13:49:14.552667 21587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:38173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:49:14.573767 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34655
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:14.904390 21782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:14.904716 21782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:14.904844 21782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:14.912461 21782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:14.912624 21782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:14.912786 21782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:49:14.923067 21782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:14.924512 21782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:14.925814 21782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:14.938894 21787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:14.939145 21788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:14.940296 21790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:14.940578 21782 server_base.cc:1034] running on GCE node
I20260927 13:49:14.941627 21782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:14.942720 21782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:14.943945 21782 hybrid_clock.cc:648] HybridClock initialized: now 1790516954943894 us; error 51 us; skew 500 ppm
I20260927 13:49:14.944285 21782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:14 dist-test-slave-q25r krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516954, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:14.952183 21782 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:49:14.954010 21782 webserver.cc:460] Webserver started at http://127.20.133.66:38905/ using document root <none> and password file <none>
I20260927 13:49:14.954734 21782 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:14.954849 21782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:14.955137 21782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:14.958451 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "6aa1b2475ff9484787a170b1cddccb46"
format_stamp: "Formatted at 2026-09-27 13:49:14 on dist-test-slave-q25r"
server_key: "9d4e8db5be2ad1896f5a275ad9c4ab3c"
server_key_iv: "ed56aa83228a7e5c50ffe5674f1a8eea"
server_key_version: "encryptionkey@0"
I20260927 13:49:14.959499 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "6aa1b2475ff9484787a170b1cddccb46"
format_stamp: "Formatted at 2026-09-27 13:49:14 on dist-test-slave-q25r"
server_key: "9d4e8db5be2ad1896f5a275ad9c4ab3c"
server_key_iv: "ed56aa83228a7e5c50ffe5674f1a8eea"
server_key_version: "encryptionkey@0"
I20260927 13:49:14.965929 21782 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 13:49:14.969599 21797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:14.970995 21782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:49:14.971194 21782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "6aa1b2475ff9484787a170b1cddccb46"
format_stamp: "Formatted at 2026-09-27 13:49:14 on dist-test-slave-q25r"
server_key: "9d4e8db5be2ad1896f5a275ad9c4ab3c"
server_key_iv: "ed56aa83228a7e5c50ffe5674f1a8eea"
server_key_version: "encryptionkey@0"
I20260927 13:49:14.971400 21782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:14.994678 21782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:14.999399 21782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:14.999662 21782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:15.000638 21782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:15.002029 21782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:15.002106 21782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:15.002189 21782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:15.002233 21782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:15.035364 21782 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:39083
I20260927 13:49:15.035400 21909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:39083 every 8 connection(s)
I20260927 13:49:15.036799 21782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:49:15.042872 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 21782
I20260927 13:49:15.043046 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:49:15.043548 21013 external_mini_cluster.cc:1428] Setting key b764a79f9400fba345700d70f3ee8116
Sep 27 13:49:15 dist-test-slave-q25r krb5kdc[21542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516954, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:15.059181 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.039438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:32835 (local address 127.20.133.126:34655)
0927 13:49:15.039622 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:15.039626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:15.040877 (+ 1251us) server_negotiation.cc:407] Connection header received
0927 13:49:15.042877 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:15.042884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:15.042982 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:15.043091 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:15.044831 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.045561 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.047175 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.047602 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.047713 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.051605 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:15.051612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:15.051618 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:15.051664 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:15.054991 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:15.055980 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:15.055987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:15.055991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:15.056163 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:15.056681 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:15.056686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:15.056689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:15.057104 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:15.057277 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:15.058556 (+ 1279us) server_negotiation.cc:299] Negotiation successful
0927 13:49:15.058819 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
I20260927 13:49:15.060202 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.039874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34655 (local address 127.20.133.66:32835)
0927 13:49:15.040703 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:15.040748 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:15.042543 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:15.043352 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:15.043362 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:15.043966 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:15.044643 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.044669 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.045727 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.045731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.047009 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.047023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.047849 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.047857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.048093 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.048787 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:15.048811 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:15.051288 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:15.055218 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:15.055236 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:15.055254 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:15.055804 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:15.056290 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:15.056293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:15.056296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:15.056547 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:15.057392 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:15.057405 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:15.058082 (+ 677us) client_negotiation.cc:769] Sending connection context
0927 13:49:15.058389 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 13:49:15.058872 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":310,"threads_started":1}
I20260927 13:49:15.061991 21910 heartbeater.cc:347] Connected to a master server at 127.20.133.126:34655
I20260927 13:49:15.062356 21910 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
I20260927 13:49:15.063205 21910 heartbeater.cc:511] Master 127.20.133.126:34655 requested a full tablet report, sending...
I20260927 13:49:15.064987 21587 ts_manager.cc:194] Registered new tserver with Master: 6aa1b2475ff9484787a170b1cddccb46 (127.20.133.66:39083)
I20260927 13:49:15.065816 21587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:32835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:49:15.084619 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34655
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:15.410012 21915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:15.410307 21915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:15.410434 21915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:15.418408 21915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:15.418550 21915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:15.418711 21915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:49:15.428895 21915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:15.430239 21915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:15.431561 21915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:15.444994 21921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:15.445305 21923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:15.445348 21920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:15.446398 21915 server_base.cc:1034] running on GCE node
I20260927 13:49:15.447062 21915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:15.448101 21915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:15.449335 21915 hybrid_clock.cc:648] HybridClock initialized: now 1790516955449257 us; error 72 us; skew 500 ppm
I20260927 13:49:15.449697 21915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:15 dist-test-slave-q25r krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516955, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:15.453184 21915 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:49:15.455060 21915 webserver.cc:460] Webserver started at http://127.20.133.67:40387/ using document root <none> and password file <none>
I20260927 13:49:15.456034 21915 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:15.456188 21915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:15.456562 21915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:15.459779 21915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf"
format_stamp: "Formatted at 2026-09-27 13:49:15 on dist-test-slave-q25r"
server_key: "85a12fbc43aaa6be5dbc0db922b0e9f4"
server_key_iv: "de619974b13576b91aa239eb83138c02"
server_key_version: "encryptionkey@0"
I20260927 13:49:15.460703 21915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf"
format_stamp: "Formatted at 2026-09-27 13:49:15 on dist-test-slave-q25r"
server_key: "85a12fbc43aaa6be5dbc0db922b0e9f4"
server_key_iv: "de619974b13576b91aa239eb83138c02"
server_key_version: "encryptionkey@0"
I20260927 13:49:15.466711 21915 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 13:49:15.470850 21930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:15.472301 21915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 13:49:15.472517 21915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf"
format_stamp: "Formatted at 2026-09-27 13:49:15 on dist-test-slave-q25r"
server_key: "85a12fbc43aaa6be5dbc0db922b0e9f4"
server_key_iv: "de619974b13576b91aa239eb83138c02"
server_key_version: "encryptionkey@0"
I20260927 13:49:15.472736 21915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:15.495913 21915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:15.500376 21915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:15.500664 21915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:15.501672 21915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:15.503072 21915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:15.503154 21915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:15.503276 21915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:15.503345 21915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:15.537038 21915 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:41697
I20260927 13:49:15.537115 22042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:41697 every 8 connection(s)
I20260927 13:49:15.538424 21915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:49:15.541927 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 21915
I20260927 13:49:15.542119 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:49:15.542536 21013 external_mini_cluster.cc:1428] Setting key af8b059669808c9477962793089ac3de
Sep 27 13:49:15 dist-test-slave-q25r krb5kdc[21542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516955, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:15.556082 21776 heartbeater.cc:503] Master 127.20.133.126:34655 was elected leader, sending a full tablet report...
I20260927 13:49:15.559984 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.541029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:38201 (local address 127.20.133.126:34655)
0927 13:49:15.541244 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:15.541248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:15.542796 (+ 1548us) server_negotiation.cc:407] Connection header received
0927 13:49:15.545357 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:15.545362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:15.545436 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:15.545524 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:15.547327 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.548095 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.549781 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.550190 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.550282 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.553771 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:15.553776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:15.553780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:15.553816 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:15.556463 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:15.557465 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:15.557472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:15.557476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:15.557661 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:15.558162 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:15.558168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:15.558172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:15.558467 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:15.558603 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:15.559408 (+ 805us) server_negotiation.cc:299] Negotiation successful
0927 13:49:15.559641 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 13:49:15.561137 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.541592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34655 (local address 127.20.133.67:38201)
0927 13:49:15.542591 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:15.542648 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:15.544989 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:15.545866 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:15.545878 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:15.546501 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:15.547133 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.547154 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.548271 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.548275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.549658 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.549671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.550368 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.550373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.550572 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.551239 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:15.551265 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:15.553495 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:15.556666 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:15.556677 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:15.556696 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:15.557307 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:15.557783 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:15.557788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:15.557792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:15.558025 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:15.558611 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:15.558619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:15.559021 (+ 402us) client_negotiation.cc:769] Sending connection context
0927 13:49:15.559277 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 13:49:15.559834 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":414,"threads_started":1}
I20260927 13:49:15.562482 22043 heartbeater.cc:347] Connected to a master server at 127.20.133.126:34655
I20260927 13:49:15.562774 22043 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:15.563516 22043 heartbeater.cc:511] Master 127.20.133.126:34655 requested a full tablet report, sending...
I20260927 13:49:15.565001 21587 ts_manager.cc:194] Registered new tserver with Master: ea1c0fb7d3b34eba94e7df9d73d94dcf (127.20.133.67:41697)
I20260927 13:49:15.565858 21587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:38201
I20260927 13:49:15.571918 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:49:15.591154 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.578635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.20.133.126:34655)
0927 13:49:15.578814 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:15.578820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:15.579284 (+ 464us) server_negotiation.cc:407] Connection header received
0927 13:49:15.579451 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:15.579455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:15.579521 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:15.579606 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:15.581194 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.581940 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.583209 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.583676 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.583795 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.586112 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:15.586117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:15.586120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:15.586156 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:15.588583 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:15.589242 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:15.589247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:15.589251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:15.589386 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:15.589738 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:15.589743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:15.589746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:15.590069 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:15.590311 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:15.590624 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 13:49:15.590834 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:49:15.597750 21587 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:49:15.604991 21587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:49:15.629032 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.617793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38855 (local address 127.0.0.1:44720)
0927 13:49:15.618626 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:15.618678 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:15.618856 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:15.622021 (+ 3165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:15.622026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:15.622066 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:15.622428 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.622440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.624207 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.624212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.626175 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.626188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.627932 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.627940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.628126 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.628178 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 13:49:15.628319 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 13:49:15.628556 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":24,"spinlock_wait_cycles":4352,"thread_start_us":440,"threads_started":1}
I20260927 13:49:15.629181 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.618482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.20.133.66:39083)
0927 13:49:15.621065 (+ 2583us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:15.621075 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:15.621103 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:49:15.621200 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:15.621209 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:15.621469 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:15.621680 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:15.622953 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.624259 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.626632 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.628042 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.628180 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.628392 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:15.628696 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 13:49:15.628870 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2423,"thread_start_us":2925,"threads_started":1}
I20260927 13:49:15.629102 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.617935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:39083 (local address 127.0.0.1:37534)
0927 13:49:15.620780 (+ 2845us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:15.620806 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:15.620929 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:15.621938 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:15.621947 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:15.621981 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:15.622770 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.622781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.624415 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.624420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.626309 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.626327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.628303 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.628310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.628508 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.628539 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 13:49:15.628624 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 13:49:15.628748 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2705,"mutex_wait_us":10,"thread_start_us":2553,"threads_started":1}
I20260927 13:49:15.630048 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.618134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44720 (local address 127.20.133.65:38855)
0927 13:49:15.621462 (+ 3328us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:15.621469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:15.621496 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:49:15.621592 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:15.621600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:15.621817 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:15.622329 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:15.622560 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.624069 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.626352 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.627732 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.627858 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.628133 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:15.629552 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0927 13:49:15.629731 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3183,"thread_start_us":3067,"threads_started":1}
I20260927 13:49:15.630326 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.620830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:41697 (local address 127.0.0.1:56992)
0927 13:49:15.621940 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:15.621960 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:15.622064 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:15.623167 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:15.623172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:15.623195 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:15.623618 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.623630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.625788 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.625794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.627886 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.627900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.629562 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.629568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.629731 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.629763 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:49:15.629845 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 13:49:15.629969 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"spinlock_wait_cycles":768,"thread_start_us":858,"threads_started":1}
I20260927 13:49:15.630345 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.621325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.20.133.67:41697)
0927 13:49:15.622626 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:15.622634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:15.622662 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:49:15.622748 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:15.622754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:15.622951 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:15.623794 (+ 843us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:15.624039 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.625642 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.628085 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.629395 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.629516 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.629735 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:15.629900 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 13:49:15.630049 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1134,"thread_start_us":1002,"threads_started":1}
I20260927 13:49:15.633396 21978 tablet_service.cc:1467] Processing CreateTablet for tablet d3ddd1fa5dff4ac5abd5acfb81149284 (DEFAULT_TABLE table=test-table [id=55d3b51d7915429ea8c726caf7f73595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:15.633548 21845 tablet_service.cc:1467] Processing CreateTablet for tablet d3ddd1fa5dff4ac5abd5acfb81149284 (DEFAULT_TABLE table=test-table [id=55d3b51d7915429ea8c726caf7f73595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:15.634068 21711 tablet_service.cc:1467] Processing CreateTablet for tablet d3ddd1fa5dff4ac5abd5acfb81149284 (DEFAULT_TABLE table=test-table [id=55d3b51d7915429ea8c726caf7f73595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:15.634694 21978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ddd1fa5dff4ac5abd5acfb81149284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:15.634902 21845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ddd1fa5dff4ac5abd5acfb81149284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:15.635329 21711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ddd1fa5dff4ac5abd5acfb81149284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:15.647416 22061 tablet_bootstrap.cc:492] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf: Bootstrap starting.
I20260927 13:49:15.649808 22062 tablet_bootstrap.cc:492] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999: Bootstrap starting.
I20260927 13:49:15.650708 22063 tablet_bootstrap.cc:492] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46: Bootstrap starting.
I20260927 13:49:15.652781 22061 tablet_bootstrap.cc:654] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:15.653842 22061 log.cc:824] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:15.654579 22062 tablet_bootstrap.cc:654] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:15.654659 22063 tablet_bootstrap.cc:654] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:15.655932 22062 log.cc:824] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:15.655932 22063 log.cc:824] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:15.656872 22061 tablet_bootstrap.cc:492] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf: No bootstrap required, opened a new log
I20260927 13:49:15.657244 22061 ts_tablet_manager.cc:1397] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 13:49:15.658891 22062 tablet_bootstrap.cc:492] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999: No bootstrap required, opened a new log
I20260927 13:49:15.659276 22063 tablet_bootstrap.cc:492] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46: No bootstrap required, opened a new log
I20260927 13:49:15.659312 22062 ts_tablet_manager.cc:1397] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 13:49:15.659579 22063 ts_tablet_manager.cc:1397] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 13:49:15.667502 22062 raft_consensus.cc:348] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f705b9cb70794465a2e985e5e6a2b999" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38855 } } peers { permanent_uuid: "6aa1b2475ff9484787a170b1cddccb46" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39083 } } peers { permanent_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 41697 } }
I20260927 13:49:15.667956 22062 raft_consensus.cc:374] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:15.668092 22062 raft_consensus.cc:729] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f705b9cb70794465a2e985e5e6a2b999, State: Initialized, Role: FOLLOWER
I20260927 13:49:15.668128 22061 raft_consensus.cc:348] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f705b9cb70794465a2e985e5e6a2b999" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38855 } } peers { permanent_uuid: "6aa1b2475ff9484787a170b1cddccb46" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39083 } } peers { permanent_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 41697 } }
I20260927 13:49:15.668635 22061 raft_consensus.cc:374] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:15.668735 22061 raft_consensus.cc:729] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1c0fb7d3b34eba94e7df9d73d94dcf, State: Initialized, Role: FOLLOWER
I20260927 13:49:15.668839 22062 consensus_queue.cc:260] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f705b9cb70794465a2e985e5e6a2b999" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38855 } } peers { permanent_uuid: "6aa1b2475ff9484787a170b1cddccb46" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39083 } } peers { permanent_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 41697 } }
I20260927 13:49:15.669296 22061 consensus_queue.cc:260] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f705b9cb70794465a2e985e5e6a2b999" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38855 } } peers { permanent_uuid: "6aa1b2475ff9484787a170b1cddccb46" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39083 } } peers { permanent_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 41697 } }
I20260927 13:49:15.670044 22063 raft_consensus.cc:348] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f705b9cb70794465a2e985e5e6a2b999" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38855 } } peers { permanent_uuid: "6aa1b2475ff9484787a170b1cddccb46" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39083 } } peers { permanent_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 41697 } }
I20260927 13:49:15.670537 22063 raft_consensus.cc:374] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:15.670656 22063 raft_consensus.cc:729] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa1b2475ff9484787a170b1cddccb46, State: Initialized, Role: FOLLOWER
I20260927 13:49:15.670840 22062 ts_tablet_manager.cc:1428] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 13:49:15.671280 22063 consensus_queue.cc:260] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f705b9cb70794465a2e985e5e6a2b999" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38855 } } peers { permanent_uuid: "6aa1b2475ff9484787a170b1cddccb46" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39083 } } peers { permanent_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 41697 } }
I20260927 13:49:15.674571 22043 heartbeater.cc:503] Master 127.20.133.126:34655 was elected leader, sending a full tablet report...
I20260927 13:49:15.674614 22061 ts_tablet_manager.cc:1428] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260927 13:49:15.675151 22063 ts_tablet_manager.cc:1428] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 13:49:15.676458 21910 heartbeater.cc:503] Master 127.20.133.126:34655 was elected leader, sending a full tablet report...
W20260927 13:49:15.775741 21777 tablet.cc:2369] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:49:15.788726 21911 tablet.cc:2369] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:49:15.789988 22044 tablet.cc:2369] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:49:15.892840 22069 raft_consensus.cc:482] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:49:15.893175 22069 raft_consensus.cc:504] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f705b9cb70794465a2e985e5e6a2b999" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38855 } } peers { permanent_uuid: "6aa1b2475ff9484787a170b1cddccb46" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39083 } } peers { permanent_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 41697 } }
I20260927 13:49:15.895085 22069 leader_election.cc:288] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f705b9cb70794465a2e985e5e6a2b999 (127.20.133.65:38855), ea1c0fb7d3b34eba94e7df9d73d94dcf (127.20.133.67:41697)
I20260927 13:49:15.901676 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.895605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:52069 (local address 127.20.133.65:38855)
0927 13:49:15.895804 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:15.895810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:15.896525 (+ 715us) server_negotiation.cc:407] Connection header received
0927 13:49:15.896689 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:15.896692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:15.896751 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:15.896827 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:15.897424 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.898109 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.899940 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.900743 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.900897 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.900974 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:15.901321 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 13:49:15.901442 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 13:49:15.901852 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.895460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38855 (local address 127.20.133.66:52069)
0927 13:49:15.896420 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:15.896442 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:15.896563 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:15.896866 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:15.896870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:15.896899 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:15.897267 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.897276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.898246 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.898249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.899793 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.899805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.901022 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.901027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.901167 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.901190 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 13:49:15.901267 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 13:49:15.901390 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":63,"thread_start_us":643,"threads_started":1}
I20260927 13:49:15.901949 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.895755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:33697 (local address 127.20.133.67:41697)
0927 13:49:15.895956 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:15.895961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:15.896387 (+ 426us) server_negotiation.cc:407] Connection header received
0927 13:49:15.896615 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:15.896619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:15.896680 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:15.896754 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:15.897378 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.898104 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.899933 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.900742 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.900899 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.900973 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:15.901410 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 13:49:15.901629 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 13:49:15.902237 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.895422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:41697 (local address 127.20.133.66:33697)
0927 13:49:15.896215 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:15.896240 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:15.896451 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:15.896807 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:15.896811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:15.896836 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:15.897219 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.897232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.898228 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.898233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.899772 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:15.899785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.900986 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.900994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:15.901142 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:15.901167 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 13:49:15.901248 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 13:49:15.901567 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":2,"thread_start_us":403,"threads_started":1}
I20260927 13:49:15.903469 21998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ddd1fa5dff4ac5abd5acfb81149284" candidate_uuid: "6aa1b2475ff9484787a170b1cddccb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf" is_pre_election: true
I20260927 13:49:15.903466 21731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ddd1fa5dff4ac5abd5acfb81149284" candidate_uuid: "6aa1b2475ff9484787a170b1cddccb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f705b9cb70794465a2e985e5e6a2b999" is_pre_election: true
I20260927 13:49:15.903997 21998 raft_consensus.cc:2454] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aa1b2475ff9484787a170b1cddccb46 in term 0.
I20260927 13:49:15.904110 21731 raft_consensus.cc:2454] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aa1b2475ff9484787a170b1cddccb46 in term 0.
I20260927 13:49:15.904750 21801 leader_election.cc:302] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aa1b2475ff9484787a170b1cddccb46, ea1c0fb7d3b34eba94e7df9d73d94dcf; no voters:
I20260927 13:49:15.905200 22069 raft_consensus.cc:2781] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:49:15.905347 22069 raft_consensus.cc:482] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:49:15.905452 22069 raft_consensus.cc:3037] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:15.907958 22069 raft_consensus.cc:504] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f705b9cb70794465a2e985e5e6a2b999" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38855 } } peers { permanent_uuid: "6aa1b2475ff9484787a170b1cddccb46" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39083 } } peers { permanent_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 41697 } }
I20260927 13:49:15.908800 22069 leader_election.cc:288] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [CANDIDATE]: Term 1 election: Requested vote from peers f705b9cb70794465a2e985e5e6a2b999 (127.20.133.65:38855), ea1c0fb7d3b34eba94e7df9d73d94dcf (127.20.133.67:41697)
I20260927 13:49:15.909389 21731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ddd1fa5dff4ac5abd5acfb81149284" candidate_uuid: "6aa1b2475ff9484787a170b1cddccb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f705b9cb70794465a2e985e5e6a2b999"
I20260927 13:49:15.909639 21731 raft_consensus.cc:3037] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:15.909559 21998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ddd1fa5dff4ac5abd5acfb81149284" candidate_uuid: "6aa1b2475ff9484787a170b1cddccb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf"
I20260927 13:49:15.909857 21998 raft_consensus.cc:3037] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:15.912233 21998 raft_consensus.cc:2454] T d3ddd1fa5dff4ac5abd5acfb81149284 P ea1c0fb7d3b34eba94e7df9d73d94dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aa1b2475ff9484787a170b1cddccb46 in term 1.
I20260927 13:49:15.912310 21731 raft_consensus.cc:2454] T d3ddd1fa5dff4ac5abd5acfb81149284 P f705b9cb70794465a2e985e5e6a2b999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aa1b2475ff9484787a170b1cddccb46 in term 1.
I20260927 13:49:15.912855 21801 leader_election.cc:302] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aa1b2475ff9484787a170b1cddccb46, ea1c0fb7d3b34eba94e7df9d73d94dcf; no voters:
I20260927 13:49:15.913211 22069 raft_consensus.cc:2781] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:15.913749 22069 raft_consensus.cc:686] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [term 1 LEADER]: Becoming Leader. State: Replica: 6aa1b2475ff9484787a170b1cddccb46, State: Running, Role: LEADER
I20260927 13:49:15.914417 22069 consensus_queue.cc:237] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f705b9cb70794465a2e985e5e6a2b999" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38855 } } peers { permanent_uuid: "6aa1b2475ff9484787a170b1cddccb46" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39083 } } peers { permanent_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 41697 } }
I20260927 13:49:15.919807 21587 catalog_manager.cc:5495] T d3ddd1fa5dff4ac5abd5acfb81149284 P 6aa1b2475ff9484787a170b1cddccb46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aa1b2475ff9484787a170b1cddccb46 (127.20.133.66). New cstate: current_term: 1 leader_uuid: "6aa1b2475ff9484787a170b1cddccb46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f705b9cb70794465a2e985e5e6a2b999" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aa1b2475ff9484787a170b1cddccb46" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea1c0fb7d3b34eba94e7df9d73d94dcf" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 41697 } health_report { overall_health: UNKNOWN } } }
Sep 27 13:49:15 dist-test-slave-q25r krb5kdc[21542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:16.006402 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:15.991218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.20.133.65:38855)
0927 13:49:15.991495 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:15.991502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:15.991965 (+ 463us) server_negotiation.cc:407] Connection header received
0927 13:49:15.992150 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:15.992154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:15.992226 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:15.992324 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:15.993856 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.994694 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.995952 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:15.996433 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:15.996550 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:16.000639 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:16.000655 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:16.000671 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:16.000717 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:16.003604 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.004366 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.004376 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.004379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.004527 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.004886 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.004890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.004892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.005249 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:16.005556 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:16.005837 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 13:49:16.006056 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
Sep 27 13:49:16 dist-test-slave-q25r krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516956, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:49:16 dist-test-slave-q25r krb5kdc[21542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516956, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:16.059525 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:16.043916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.20.133.65:38855)
0927 13:49:16.044163 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:16.044171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:16.044676 (+ 505us) server_negotiation.cc:407] Connection header received
0927 13:49:16.044860 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:16.044866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:16.044956 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:16.045096 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:16.046681 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:16.047755 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:16.049264 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:16.049689 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:16.049802 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:16.054156 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:16.054161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:16.054165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:16.054203 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:16.057090 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.057811 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.057815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.057817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.057952 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.058340 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.058343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.058345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.058606 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:16.058796 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:16.059071 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 13:49:16.059264 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 13:49:16 dist-test-slave-q25r krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:49:16 dist-test-slave-q25r krb5kdc[21542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:16.110522 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:16.095736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.20.133.65:38855)
0927 13:49:16.095984 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:16.095991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:16.096432 (+ 441us) server_negotiation.cc:407] Connection header received
0927 13:49:16.096646 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:16.096650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:16.096719 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:16.096816 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:16.098150 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:16.098834 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:16.100059 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:16.100525 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:16.100635 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:16.105004 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:16.105011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:16.105015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:16.105056 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:16.107525 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.108522 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.108526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.108529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.108664 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.109099 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.109104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.109108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.109438 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:16.109630 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:16.109959 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 13:49:16.110207 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 13:49:16.113042 21691 tablet_service.cc:2908] Rejecting scan request for tablet d3ddd1fa5dff4ac5abd5acfb81149284: Uninitialized: clean time has not yet been initialized
I20260927 13:49:16.116397 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 21648
I20260927 13:49:16.130610 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:16.117467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.20.133.65:38855)
0927 13:49:16.117710 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:16.117717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:16.118010 (+ 293us) server_negotiation.cc:407] Connection header received
0927 13:49:16.118176 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:16.118189 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:16.118260 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:16.118353 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:16.119587 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:16.120279 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:16.121617 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:16.122043 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:16.122155 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:16.124650 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:16.124656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:16.124659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:16.124788 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:16.127404 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.128261 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.128267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.128270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.128498 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.128941 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.128946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.128949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.129309 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:16.129687 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:16.129970 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 13:49:16.130274 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 13:49:16.131522 21771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:16.224220 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 21648
I20260927 13:49:16.238514 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 21782
Sep 27 13:49:16 dist-test-slave-q25r krb5kdc[21542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:49:16.257861 22086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:16.239481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.20.133.66:39083)
0927 13:49:16.240046 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:16.240052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:16.240070 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:49:16.240139 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:16.240144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:16.240228 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:16.240373 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:16.242323 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:16.243103 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:16.245082 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:16.245850 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:16.246074 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:16.251633 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:16.251654 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:16.251671 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:16.251719 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:16.254698 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.255474 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.255481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.255484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.255634 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.256054 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.256057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.256059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.256494 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:16.256775 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:16.257262 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 13:49:16.257490 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":273,"threads_started":1}
I20260927 13:49:16.258702 21905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:16.357486 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 21782
W20260927 13:49:16.372823 21933 connection.cc:535] server connection from 127.20.133.66:33697 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 13:49:16.373189 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 21915
Sep 27 13:49:16 dist-test-slave-q25r krb5kdc[21542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:49:16.390128 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:16.374055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.20.133.67:41697)
0927 13:49:16.374297 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:16.374302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:16.374324 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:49:16.374502 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:16.374507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:16.374598 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:16.374702 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:16.376028 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:16.377028 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:16.378402 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:16.378950 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:16.379064 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:16.383561 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:16.383581 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:16.383602 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:16.383649 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:16.386845 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.387679 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.387688 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.387691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.387892 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:16.388231 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:16.388235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:16.388238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:16.389038 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:16.389305 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:16.389561 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 13:49:16.389776 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 13:49:16.390976 22038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:16.485958 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 21915
I20260927 13:49:16.499264 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 21555
I20260927 13:49:16.500684 21617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:16.614670 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 21555
2026-09-27T13:49:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4468 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:49:16 dist-test-slave-q25r krb5kdc[22092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:49:16 dist-test-slave-q25r krb5kdc[22092](info): set up 2 sockets
Sep 27 13:49:16 dist-test-slave-q25r krb5kdc[22092](info): commencing operation
krb5kdc: starting...
W20260927 13:49:17.636754 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:49:17 dist-test-slave-q25r krb5kdc[22092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:49:17.839955 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:34807
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:34807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:18.182515 22105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:18.182863 22105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:18.183002 22105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:18.190812 22105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:18.190951 22105 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 13:49:18.190999 22105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:18.191103 22105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:18.191188 22105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:18.191285 22105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:18.201896 22105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:34807
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:34807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:18.203347 22105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:18.204815 22105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:18.217146 22110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:18.217108 22111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:18.217171 22113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:18.218155 22105 server_base.cc:1034] running on GCE node
I20260927 13:49:18.218976 22105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:18.220454 22105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:18.221684 22105 hybrid_clock.cc:648] HybridClock initialized: now 1790516958221623 us; error 35 us; skew 500 ppm
I20260927 13:49:18.221997 22105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:18 dist-test-slave-q25r krb5kdc[22092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:18.225096 22105 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:49:18.226639 22105 webserver.cc:460] Webserver started at http://127.20.133.126:33075/ using document root <none> and password file <none>
I20260927 13:49:18.227329 22105 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:18.227419 22105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:18.227690 22105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:18.230831 22105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "5cae2f07f7ac425f98d16ee034d548fe"
format_stamp: "Formatted at 2026-09-27 13:49:18 on dist-test-slave-q25r"
server_key: "6820be2bb7936ba8afd757f08be3e201"
server_key_iv: "4167773ed46785dc10cbf92e21db5892"
server_key_version: "encryptionkey@0"
I20260927 13:49:18.231824 22105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "5cae2f07f7ac425f98d16ee034d548fe"
format_stamp: "Formatted at 2026-09-27 13:49:18 on dist-test-slave-q25r"
server_key: "6820be2bb7936ba8afd757f08be3e201"
server_key_iv: "4167773ed46785dc10cbf92e21db5892"
server_key_version: "encryptionkey@0"
I20260927 13:49:18.237375 22105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 13:49:18.243225 22120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:18.245515 22105 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260927 13:49:18.245733 22105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "5cae2f07f7ac425f98d16ee034d548fe"
format_stamp: "Formatted at 2026-09-27 13:49:18 on dist-test-slave-q25r"
server_key: "6820be2bb7936ba8afd757f08be3e201"
server_key_iv: "4167773ed46785dc10cbf92e21db5892"
server_key_version: "encryptionkey@0"
I20260927 13:49:18.245955 22105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:18.275666 22105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:18.280555 22105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:18.280872 22105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:18.303015 22105 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:34807
I20260927 13:49:18.303050 22181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:34807 every 8 connection(s)
I20260927 13:49:18.304641 22105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:18.308971 22182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:18.310421 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 22105
I20260927 13:49:18.310665 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:18.311457 21013 external_mini_cluster.cc:1428] Setting key 420a94019db9418285fd7ddaa1c9c82b
I20260927 13:49:18.323928 22182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe: Bootstrap starting.
Sep 27 13:49:18 dist-test-slave-q25r krb5kdc[22092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:18.327131 22182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:18.328325 22182 log.cc:824] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:18.331622 22182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe: No bootstrap required, opened a new log
I20260927 13:49:18.336071 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:18.313618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.20.133.126:34807)
0927 13:49:18.314459 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:18.314474 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:18.314516 (+ 42us) server_negotiation.cc:407] Connection header received
0927 13:49:18.316188 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:18.316211 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:18.316710 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:18.317055 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:18.318829 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:18.320939 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:18.322757 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:18.323475 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:18.323658 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:18.327852 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:18.327863 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:18.327882 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:18.327940 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:18.331455 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:18.332449 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:18.332457 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:18.332463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:18.332672 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:18.333129 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:18.333133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:18.333135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:18.333776 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:18.334051 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:18.334537 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 13:49:18.334973 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":362,"threads_started":1}
I20260927 13:49:18.340080 22182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cae2f07f7ac425f98d16ee034d548fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34807 } }
I20260927 13:49:18.340482 22182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:18.340587 22182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cae2f07f7ac425f98d16ee034d548fe, State: Initialized, Role: FOLLOWER
I20260927 13:49:18.341141 22182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cae2f07f7ac425f98d16ee034d548fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34807 } }
I20260927 13:49:18.341396 22182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:18.341524 22182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:18.341674 22182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:18.344298 22182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cae2f07f7ac425f98d16ee034d548fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34807 } }
I20260927 13:49:18.344883 22182 leader_election.cc:302] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cae2f07f7ac425f98d16ee034d548fe; no voters:
I20260927 13:49:18.345463 22182 leader_election.cc:288] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:18.345606 22187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:18.346231 22187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [term 1 LEADER]: Becoming Leader. State: Replica: 5cae2f07f7ac425f98d16ee034d548fe, State: Running, Role: LEADER
I20260927 13:49:18.346791 22187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cae2f07f7ac425f98d16ee034d548fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34807 } }
I20260927 13:49:18.347594 22182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:18.349817 22189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cae2f07f7ac425f98d16ee034d548fe. Latest consensus state: current_term: 1 leader_uuid: "5cae2f07f7ac425f98d16ee034d548fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cae2f07f7ac425f98d16ee034d548fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34807 } } }
I20260927 13:49:18.349804 22188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cae2f07f7ac425f98d16ee034d548fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cae2f07f7ac425f98d16ee034d548fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34807 } } }
I20260927 13:49:18.350234 22189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:18.350272 22188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:18.351608 22194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:18.355969 22194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:18.366386 22194 catalog_manager.cc:1347] Generated new cluster ID: d656f64d53614fa190b2cd6cb5aff89f
I20260927 13:49:18.366514 22194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:18.375533 22194 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:18.377550 22194 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:18.387485 22194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe: Generated new TSK 0
I20260927 13:49:18.388691 22194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:49:18.439473 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34807
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:18.783877 22208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:18.784184 22208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:18.784354 22208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:18.792025 22208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:18.792145 22208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:18.792279 22208 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:49:18.792424 22208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:49:18.802584 22208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.133.126:34807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:18.803946 22208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:18.805393 22208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:18.819193 22213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:18.819147 22216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:18.819213 22214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:18.820030 22208 server_base.cc:1034] running on GCE node
I20260927 13:49:18.820711 22208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:18.821676 22208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:18.822907 22208 hybrid_clock.cc:648] HybridClock initialized: now 1790516958822830 us; error 51 us; skew 500 ppm
I20260927 13:49:18.823184 22208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:18 dist-test-slave-q25r krb5kdc[22092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:18.826470 22208 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:49:18.827967 22208 webserver.cc:460] Webserver started at http://127.20.133.65:40125/ using document root <none> and password file <none>
I20260927 13:49:18.828740 22208 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:18.828831 22208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:18.829159 22208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:18.832158 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "d8f43c3d9d104b32aa515e06d8130c69"
format_stamp: "Formatted at 2026-09-27 13:49:18 on dist-test-slave-q25r"
server_key: "e80138c142d11534f4c823bef7f4e9f9"
server_key_iv: "d11bfda93a3e70bc66167979c0880cd9"
server_key_version: "encryptionkey@0"
I20260927 13:49:18.832985 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "d8f43c3d9d104b32aa515e06d8130c69"
format_stamp: "Formatted at 2026-09-27 13:49:18 on dist-test-slave-q25r"
server_key: "e80138c142d11534f4c823bef7f4e9f9"
server_key_iv: "d11bfda93a3e70bc66167979c0880cd9"
server_key_version: "encryptionkey@0"
I20260927 13:49:18.838065 22208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 13:49:18.841845 22223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:18.843894 22208 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:49:18.844075 22208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "d8f43c3d9d104b32aa515e06d8130c69"
format_stamp: "Formatted at 2026-09-27 13:49:18 on dist-test-slave-q25r"
server_key: "e80138c142d11534f4c823bef7f4e9f9"
server_key_iv: "d11bfda93a3e70bc66167979c0880cd9"
server_key_version: "encryptionkey@0"
I20260927 13:49:18.844405 22208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:18.862824 22208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:18.867195 22208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:18.867470 22208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:18.870453 22208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:18.870590 22208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:18.870748 22208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:18.870826 22208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:49:18 dist-test-slave-q25r krb5kdc[22092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:18.894146 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:18.870831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:59131 (local address 127.20.133.126:34807)
0927 13:49:18.871435 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:18.871441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:18.872591 (+ 1150us) server_negotiation.cc:407] Connection header received
0927 13:49:18.874666 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:18.874673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:18.874764 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:18.874907 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:18.877016 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:18.877913 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:18.881366 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:18.882099 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:18.882270 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:18.887155 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:18.887161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:18.887167 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:18.887211 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:18.889706 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:18.890864 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:18.890871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:18.890874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:18.891003 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:18.891702 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:18.891708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:18.891711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:18.892059 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:18.892204 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:18.893511 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0927 13:49:18.893784 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":236,"threads_started":1}
I20260927 13:49:18.895697 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:18.871539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34807 (local address 127.20.133.65:59131)
0927 13:49:18.872436 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:18.872481 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:18.874336 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:18.875089 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:18.875097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:18.875810 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:18.876797 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:18.876826 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:18.878153 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:18.878160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:18.881033 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:18.881056 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:18.882486 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:18.882499 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:18.882909 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:18.883913 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:18.883951 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:18.886831 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:18.889884 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:18.889896 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:18.889919 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:18.890665 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:18.891174 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:18.891180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:18.891184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:18.891527 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:18.892205 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:18.892214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:18.892986 (+ 772us) client_negotiation.cc:769] Sending connection context
0927 13:49:18.893401 (+ 415us) client_negotiation.cc:240] Negotiation successful
0927 13:49:18.894029 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":397,"threads_started":1}
I20260927 13:49:18.918916 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:18.903915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:51333 (local address 127.20.133.126:34807)
0927 13:49:18.904098 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:18.904104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:18.904387 (+ 283us) server_negotiation.cc:407] Connection header received
0927 13:49:18.904589 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:18.904592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:18.904663 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:18.904740 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:18.906068 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:18.906763 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:18.908922 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:18.909408 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:18.909553 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:18.912207 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:18.912214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:18.912219 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:18.912273 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:18.915604 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:18.916727 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:18.916734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:18.916737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:18.916929 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:18.917426 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:18.917432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:18.917435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:18.917737 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:18.917893 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:18.918335 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 13:49:18.918566 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:49:18.919020 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:18.903430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34807 (local address 127.20.133.65:51333)
0927 13:49:18.904206 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:18.904239 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:18.904450 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:18.904804 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:18.904810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:18.905475 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:18.905924 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:18.905940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:18.906894 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:18.906900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:18.908727 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:18.908747 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:18.909647 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:18.909656 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:18.909849 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:18.910536 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:18.910565 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:18.911939 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:18.915805 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:18.915812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:18.915821 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:18.916518 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:18.917086 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:18.917092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:18.917096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:18.917271 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:18.917866 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:18.917870 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:18.918056 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 13:49:18.918176 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 13:49:18.918564 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":532,"threads_started":1}
I20260927 13:49:18.921401 22208 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:33077
I20260927 13:49:18.921423 22343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:33077 every 8 connection(s)
I20260927 13:49:18.922672 22208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:49:18.924465 22344 heartbeater.cc:347] Connected to a master server at 127.20.133.126:34807
I20260927 13:49:18.924647 22344 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:18.925211 22344 heartbeater.cc:511] Master 127.20.133.126:34807 requested a full tablet report, sending...
I20260927 13:49:18.926976 22137 ts_manager.cc:194] Registered new tserver with Master: d8f43c3d9d104b32aa515e06d8130c69 (127.20.133.65:33077)
I20260927 13:49:18.928396 22137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:59131
I20260927 13:49:18.932652 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 22208
I20260927 13:49:18.932852 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:49:18.933420 21013 external_mini_cluster.cc:1428] Setting key c22b12eb68fb3f1edee20994dddec3d3
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:49:18.971809 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34807
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:19.289243 22348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:19.289590 22348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:19.289710 22348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:19.297413 22348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:19.297555 22348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:19.297689 22348 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:49:19.297791 22348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:49:19.308284 22348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.133.126:34807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:19.309697 22348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:19.311076 22348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:19.324572 22354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:19.324580 22353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:19.328630 22356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:19.332501 22348 server_base.cc:1034] running on GCE node
I20260927 13:49:19.333302 22348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:19.334360 22348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:19.335647 22348 hybrid_clock.cc:648] HybridClock initialized: now 1790516959335575 us; error 38 us; skew 500 ppm
I20260927 13:49:19.335979 22348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:19 dist-test-slave-q25r krb5kdc[22092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:19.339275 22348 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:49:19.340904 22348 webserver.cc:460] Webserver started at http://127.20.133.66:42685/ using document root <none> and password file <none>
I20260927 13:49:19.341709 22348 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:19.341817 22348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:19.342136 22348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:19.345250 22348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "789c301090a244dfa089fd5861180ab5"
format_stamp: "Formatted at 2026-09-27 13:49:19 on dist-test-slave-q25r"
server_key: "c2bb600dde4ba94dd85af04812f9a80a"
server_key_iv: "4c0da7d50e1346d3a7aba2ca1784000e"
server_key_version: "encryptionkey@0"
I20260927 13:49:19.346062 22348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "789c301090a244dfa089fd5861180ab5"
format_stamp: "Formatted at 2026-09-27 13:49:19 on dist-test-slave-q25r"
server_key: "c2bb600dde4ba94dd85af04812f9a80a"
server_key_iv: "4c0da7d50e1346d3a7aba2ca1784000e"
server_key_version: "encryptionkey@0"
I20260927 13:49:19.352036 22348 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 13:49:19.356302 22363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:19.357885 22348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 13:49:19.358057 22348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "789c301090a244dfa089fd5861180ab5"
format_stamp: "Formatted at 2026-09-27 13:49:19 on dist-test-slave-q25r"
server_key: "c2bb600dde4ba94dd85af04812f9a80a"
server_key_iv: "4c0da7d50e1346d3a7aba2ca1784000e"
server_key_version: "encryptionkey@0"
I20260927 13:49:19.358294 22348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:19.371538 22348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:19.377604 22348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:19.377970 22348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:19.382094 22348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:19.382227 22348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:19.382385 22348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:19.382445 22348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:49:19 dist-test-slave-q25r krb5kdc[22092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:19.409510 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:19.382255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:36591 (local address 127.20.133.126:34807)
0927 13:49:19.382434 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:19.382439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:19.384053 (+ 1614us) server_negotiation.cc:407] Connection header received
0927 13:49:19.386913 (+ 2860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:19.386919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:19.387012 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:19.387144 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:19.389778 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.390767 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.393928 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.394561 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.394708 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:19.399930 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:19.399936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:19.399940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:19.399981 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:19.403597 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:19.404492 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:19.404497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:19.404499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:19.404655 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:19.405347 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:19.405353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:19.405356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:19.405791 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:19.405961 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:19.408880 (+ 2919us) server_negotiation.cc:299] Negotiation successful
0927 13:49:19.409167 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 13:49:19.409652 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:19.382968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34807 (local address 127.20.133.66:36591)
0927 13:49:19.383802 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:19.383869 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:19.386505 (+ 2636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:19.387472 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:19.387486 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:19.388324 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:19.389469 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:19.389506 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.390992 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.390999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:19.393690 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:19.393713 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.394854 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.394862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:19.395221 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:19.396180 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:19.396225 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:19.399595 (+ 3370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:19.403785 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:19.403793 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:19.403810 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:19.404302 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:19.404786 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:19.404793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:19.404797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:19.405162 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:19.406061 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:19.406072 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:19.406700 (+ 628us) client_negotiation.cc:769] Sending connection context
0927 13:49:19.407109 (+ 409us) client_negotiation.cc:240] Negotiation successful
0927 13:49:19.407868 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":324,"threads_started":1}
I20260927 13:49:19.426391 22348 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:38119
I20260927 13:49:19.426465 22482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:38119 every 8 connection(s)
I20260927 13:49:19.427619 22348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:49:19.429579 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 22348
I20260927 13:49:19.429775 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:49:19.430022 22483 heartbeater.cc:347] Connected to a master server at 127.20.133.126:34807
I20260927 13:49:19.430250 22483 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:19.430337 21013 external_mini_cluster.cc:1428] Setting key e8914a27f4618367f270da6238d38220
I20260927 13:49:19.431088 22483 heartbeater.cc:511] Master 127.20.133.126:34807 requested a full tablet report, sending...
I20260927 13:49:19.432178 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:19.417453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:41487 (local address 127.20.133.126:34807)
0927 13:49:19.417650 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:19.417655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:19.418392 (+ 737us) server_negotiation.cc:407] Connection header received
0927 13:49:19.418744 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:19.418751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:19.418844 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:19.418953 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:19.420173 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.421145 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.423113 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.423717 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.423842 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:19.426105 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:19.426110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:19.426113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:19.426163 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:19.429197 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:19.430031 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:19.430037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:19.430041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:19.430215 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:19.430613 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:19.430618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:19.430622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:19.431007 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:19.431181 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:19.431633 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 13:49:19.431857 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:49:19.432289 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:19.417460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34807 (local address 127.20.133.66:41487)
0927 13:49:19.418272 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:19.418293 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:19.418533 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:19.418992 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:19.418997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:19.419625 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:19.420010 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:19.420023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.421334 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.421338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:19.422932 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:19.422950 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.424096 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.424104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:19.424303 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:19.424952 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:19.424969 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:19.425899 (+ 930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:19.429359 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:19.429364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:19.429368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:19.429875 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:19.430329 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:19.430334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:19.430337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:19.430502 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:19.431146 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:19.431151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:19.431355 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 13:49:19.431515 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 13:49:19.431865 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":452,"threads_started":1}
I20260927 13:49:19.432978 22137 ts_manager.cc:194] Registered new tserver with Master: 789c301090a244dfa089fd5861180ab5 (127.20.133.66:38119)
I20260927 13:49:19.434064 22137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:36591
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:49:19.472931 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34807
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:19.793995 22487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:19.794301 22487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:19.794428 22487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:19.801878 22487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:19.802026 22487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:19.802157 22487 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:49:19.802269 22487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:49:19.813804 22487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.133.126:34807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:19.815285 22487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:19.816645 22487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:19.829344 22492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:19.829362 22493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:19.829438 22495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:19.830569 22487 server_base.cc:1034] running on GCE node
I20260927 13:49:19.831182 22487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:19.832113 22487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:19.833331 22487 hybrid_clock.cc:648] HybridClock initialized: now 1790516959833252 us; error 52 us; skew 500 ppm
I20260927 13:49:19.833657 22487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:19 dist-test-slave-q25r krb5kdc[22092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:19.836681 22487 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:49:19.838037 22487 webserver.cc:460] Webserver started at http://127.20.133.67:44321/ using document root <none> and password file <none>
I20260927 13:49:19.838712 22487 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:19.838814 22487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:19.839131 22487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:19.842031 22487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "08c2accdaddd43579aba2260481a9c0b"
format_stamp: "Formatted at 2026-09-27 13:49:19 on dist-test-slave-q25r"
server_key: "0f4f27a3d152566d57eae44a70e2c76e"
server_key_iv: "32fa181d63dcad58d4a31cf2702dd439"
server_key_version: "encryptionkey@0"
I20260927 13:49:19.842823 22487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "08c2accdaddd43579aba2260481a9c0b"
format_stamp: "Formatted at 2026-09-27 13:49:19 on dist-test-slave-q25r"
server_key: "0f4f27a3d152566d57eae44a70e2c76e"
server_key_iv: "32fa181d63dcad58d4a31cf2702dd439"
server_key_version: "encryptionkey@0"
I20260927 13:49:19.848151 22487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:49:19.852036 22502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:19.853557 22487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:49:19.853747 22487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "08c2accdaddd43579aba2260481a9c0b"
format_stamp: "Formatted at 2026-09-27 13:49:19 on dist-test-slave-q25r"
server_key: "0f4f27a3d152566d57eae44a70e2c76e"
server_key_iv: "32fa181d63dcad58d4a31cf2702dd439"
server_key_version: "encryptionkey@0"
I20260927 13:49:19.853968 22487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:19.876700 22487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:19.881246 22487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:19.881525 22487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:19.884291 22487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:19.884419 22487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 13:49:19.884564 22487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:19.884635 22487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:49:19 dist-test-slave-q25r krb5kdc[22092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:19.911369 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:19.885119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:35859 (local address 127.20.133.126:34807)
0927 13:49:19.885334 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:19.885341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:19.887247 (+ 1906us) server_negotiation.cc:407] Connection header received
0927 13:49:19.889841 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:19.889849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:19.889967 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:19.890959 (+ 992us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:19.893621 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.894582 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.897815 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.898443 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.898587 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:19.903598 (+ 5011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:19.903603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:19.903607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:19.903644 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:19.907170 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:19.908308 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:19.908314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:19.908318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:19.908543 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:19.909255 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:19.909258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:19.909260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:19.909554 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:19.909677 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:19.910875 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0927 13:49:19.911086 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:49:19.913266 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:19.886077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34807 (local address 127.20.133.67:35859)
0927 13:49:19.887020 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:19.887071 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:19.889412 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:19.891132 (+ 1720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:19.891148 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:19.892167 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:19.893320 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:19.893360 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.894802 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.894809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:19.897522 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:19.897542 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.898792 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.898801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:19.899030 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:19.899904 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:19.899939 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:19.903287 (+ 3348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:19.907421 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:19.907432 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:19.907462 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:19.908151 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:19.908704 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:19.908711 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:19.908716 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:19.909104 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:19.909695 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:19.909707 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:19.910354 (+ 647us) client_negotiation.cc:769] Sending connection context
0927 13:49:19.910751 (+ 397us) client_negotiation.cc:240] Negotiation successful
0927 13:49:19.911439 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":419,"threads_started":1}
I20260927 13:49:19.931290 22344 heartbeater.cc:503] Master 127.20.133.126:34807 was elected leader, sending a full tablet report...
I20260927 13:49:19.934515 22487 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:43113
I20260927 13:49:19.934557 22621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:43113 every 8 connection(s)
I20260927 13:49:19.936084 22487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:49:19.938253 22622 heartbeater.cc:347] Connected to a master server at 127.20.133.126:34807
I20260927 13:49:19.938515 22622 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:19.939298 22622 heartbeater.cc:511] Master 127.20.133.126:34807 requested a full tablet report, sending...
I20260927 13:49:19.940810 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:19.925044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:48853 (local address 127.20.133.126:34807)
0927 13:49:19.925259 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:19.925265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:19.925946 (+ 681us) server_negotiation.cc:407] Connection header received
0927 13:49:19.926191 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:19.926198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:19.926292 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:19.926429 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:19.927901 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.929058 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.931240 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.931860 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.931999 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:19.934614 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:19.934621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:19.934625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:19.934663 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:19.938217 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:19.938941 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:19.938946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:19.938949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:19.939108 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:19.939483 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:19.939488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:19.939492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:19.939766 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:19.939877 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:19.940306 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 13:49:19.940538 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:49:19.940804 22137 ts_manager.cc:194] Registered new tserver with Master: 08c2accdaddd43579aba2260481a9c0b (127.20.133.67:43113)
I20260927 13:49:19.940923 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:19.925114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34807 (local address 127.20.133.67:48853)
0927 13:49:19.925810 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:19.925829 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:19.925979 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:19.926489 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:19.926493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:19.927196 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:19.927670 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:19.927689 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.929249 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.929255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:19.931073 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:19.931090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.932251 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.932258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:19.932432 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:19.933189 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:19.933225 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:19.934378 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:19.938338 (+ 3960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:19.938343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:19.938346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:19.938822 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:19.939231 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:19.939235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:19.939237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:19.939366 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:19.939888 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:19.939893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:19.940074 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 13:49:19.940201 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 13:49:19.940526 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":394,"threads_started":1}
I20260927 13:49:19.941825 22137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:35859
I20260927 13:49:19.942884 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 22487
I20260927 13:49:19.943046 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:49:19.943511 21013 external_mini_cluster.cc:1428] Setting key 25650d89fb787c477dc0ce605ac8ed44
I20260927 13:49:19.946237 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:49:19 dist-test-slave-q25r krb5kdc[22092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516959, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:49:19 dist-test-slave-q25r krb5kdc[22092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516959, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:19.995182 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:19.977950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.20.133.126:34807)
0927 13:49:19.978199 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:19.978206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:19.978320 (+ 114us) server_negotiation.cc:407] Connection header received
0927 13:49:19.978526 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:19.978530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:19.978597 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:19.978693 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:19.980089 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.980890 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.982147 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:19.982582 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:19.982704 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:19.989985 (+ 7281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:19.989992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:19.989995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:19.990040 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:19.992537 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:19.993212 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:19.993217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:19.993220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:19.993359 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:19.993743 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:19.993747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:19.993749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:19.994037 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:19.994275 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:19.994601 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 13:49:19.994839 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 13:49:19.999406 22137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:49:20.006353 22137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:49:20.034435 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.018927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:38119 (local address 127.0.0.1:59804)
0927 13:49:20.019784 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:20.019823 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:20.020000 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:20.021163 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:20.021172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:20.021205 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:20.021634 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.021646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.025056 (+ 3410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.025063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.029265 (+ 4202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.029292 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.033209 (+ 3917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.033218 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.033495 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.033550 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 13:49:20.033689 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 13:49:20.033922 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"spinlock_wait_cycles":896,"thread_start_us":542,"threads_started":1}
I20260927 13:49:20.035756 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.019308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.20.133.66:38119)
0927 13:49:20.020287 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.020296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.020323 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:49:20.020452 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.020460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.020717 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.020906 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:20.021854 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.024815 (+ 2961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.029553 (+ 4738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.032625 (+ 3072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.032786 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.033018 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.034963 (+ 1945us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.035184 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":828,"threads_started":1}
I20260927 13:49:20.039837 22413 tablet_service.cc:1467] Processing CreateTablet for tablet 34f753af3be948089fbf7ba071ba7aed (DEFAULT_TABLE table=test-table [id=b79f62c3b63d4cebb81407ff19f6e1bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:20.041455 22413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f753af3be948089fbf7ba071ba7aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:20.042074 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.020280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33077 (local address 127.0.0.1:43200)
0927 13:49:20.034668 (+ 14388us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:20.034685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:20.034778 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:20.035957 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:20.035962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:20.035988 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:20.036197 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.036209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.037979 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.037984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.039801 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.039812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.041245 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.041251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.041449 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.041481 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:49:20.041578 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 13:49:20.041734 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14307,"thread_start_us":14779,"threads_started":1}
I20260927 13:49:20.043217 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.020723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.20.133.65:33077)
0927 13:49:20.035068 (+ 14345us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.035078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.035110 (+ 32us) server_negotiation.cc:407] Connection header received
0927 13:49:20.035236 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.035244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.035526 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.035721 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:20.036397 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.037797 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.039964 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.041057 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.041188 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.041399 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.041755 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.042936 (+ 1181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14195,"thread_start_us":14069,"threads_started":1}
I20260927 13:49:20.045840 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.019627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:43113 (local address 127.0.0.1:53548)
0927 13:49:20.034390 (+ 14763us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:20.034416 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:20.034573 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:20.038915 (+ 4342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:20.038920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:20.038948 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:20.039343 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.039355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.041361 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.041366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.043389 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.043406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.045046 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.045053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.045229 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.045265 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 13:49:20.045351 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 13:49:20.045494 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14581,"thread_start_us":19034,"threads_started":1}
I20260927 13:49:20.045881 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.020179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.20.133.67:43113)
0927 13:49:20.037684 (+ 17505us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.037692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.037724 (+ 32us) server_negotiation.cc:407] Connection header received
0927 13:49:20.037821 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.037828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.038084 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.038275 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:20.039581 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.041187 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.043560 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.044833 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.044958 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.045202 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.045412 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.045572 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17328,"thread_start_us":17159,"threads_started":1}
I20260927 13:49:20.047112 22274 tablet_service.cc:1467] Processing CreateTablet for tablet 34f753af3be948089fbf7ba071ba7aed (DEFAULT_TABLE table=test-table [id=b79f62c3b63d4cebb81407ff19f6e1bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:20.048632 22274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f753af3be948089fbf7ba071ba7aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:20.049297 22552 tablet_service.cc:1467] Processing CreateTablet for tablet 34f753af3be948089fbf7ba071ba7aed (DEFAULT_TABLE table=test-table [id=b79f62c3b63d4cebb81407ff19f6e1bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:20.050652 22552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f753af3be948089fbf7ba071ba7aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:20.057154 22641 tablet_bootstrap.cc:492] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: Bootstrap starting.
I20260927 13:49:20.061281 22641 tablet_bootstrap.cc:654] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:20.062688 22641 log.cc:824] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:20.063225 22642 tablet_bootstrap.cc:492] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: Bootstrap starting.
I20260927 13:49:20.064718 22643 tablet_bootstrap.cc:492] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: Bootstrap starting.
I20260927 13:49:20.067219 22642 tablet_bootstrap.cc:654] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:20.067960 22643 tablet_bootstrap.cc:654] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:20.068543 22642 log.cc:824] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:20.069118 22643 log.cc:824] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:20.072635 22641 tablet_bootstrap.cc:492] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: No bootstrap required, opened a new log
I20260927 13:49:20.072952 22641 ts_tablet_manager.cc:1397] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20260927 13:49:20.073050 22643 tablet_bootstrap.cc:492] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: No bootstrap required, opened a new log
I20260927 13:49:20.073292 22643 ts_tablet_manager.cc:1397] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 13:49:20.074481 22642 tablet_bootstrap.cc:492] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: No bootstrap required, opened a new log
I20260927 13:49:20.074709 22642 ts_tablet_manager.cc:1397] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 13:49:20.081664 22642 raft_consensus.cc:348] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.082086 22642 raft_consensus.cc:374] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:20.082181 22642 raft_consensus.cc:729] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f43c3d9d104b32aa515e06d8130c69, State: Initialized, Role: FOLLOWER
I20260927 13:49:20.082824 22642 consensus_queue.cc:260] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.083945 22641 raft_consensus.cc:348] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.083945 22643 raft_consensus.cc:348] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.084434 22641 raft_consensus.cc:374] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:20.084427 22643 raft_consensus.cc:374] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:20.084537 22641 raft_consensus.cc:729] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789c301090a244dfa089fd5861180ab5, State: Initialized, Role: FOLLOWER
I20260927 13:49:20.084534 22643 raft_consensus.cc:729] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c2accdaddd43579aba2260481a9c0b, State: Initialized, Role: FOLLOWER
I20260927 13:49:20.084718 22642 ts_tablet_manager.cc:1428] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260927 13:49:20.085171 22643 consensus_queue.cc:260] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.085172 22641 consensus_queue.cc:260] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.086263 22622 heartbeater.cc:503] Master 127.20.133.126:34807 was elected leader, sending a full tablet report...
I20260927 13:49:20.087046 22643 ts_tablet_manager.cc:1428] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 13:49:20.088282 22483 heartbeater.cc:503] Master 127.20.133.126:34807 was elected leader, sending a full tablet report...
I20260927 13:49:20.088794 22641 ts_tablet_manager.cc:1428] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260927 13:49:20.175033 22345 tablet.cc:2369] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:49:20.177944 22647 raft_consensus.cc:482] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:49:20.178253 22647 raft_consensus.cc:504] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
W20260927 13:49:20.179726 22484 tablet.cc:2369] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:49:20.180038 22647 leader_election.cc:288] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789c301090a244dfa089fd5861180ab5 (127.20.133.66:38119), 08c2accdaddd43579aba2260481a9c0b (127.20.133.67:43113)
I20260927 13:49:20.187079 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.180576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:33285 (local address 127.20.133.67:43113)
0927 13:49:20.180784 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.180790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.181623 (+ 833us) server_negotiation.cc:407] Connection header received
0927 13:49:20.181795 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.181800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.181867 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.181935 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:20.182598 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.183458 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.185396 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.186160 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.186318 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.186391 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.186691 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.186822 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:49:20.187094 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.180483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:50279 (local address 127.20.133.66:38119)
0927 13:49:20.180701 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.180707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.181666 (+ 959us) server_negotiation.cc:407] Connection header received
0927 13:49:20.181832 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.181836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.181895 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.181980 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:20.182678 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.183482 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.185411 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.186144 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.186296 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.186377 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.186684 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.186820 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:49:20.187271 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.180798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:38119 (local address 127.20.133.65:50279)
0927 13:49:20.181544 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:20.181558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:20.181698 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:20.182011 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:20.182015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:20.182038 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:20.182510 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.182522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.183612 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.183615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.185249 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.185261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.186375 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.186385 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.186522 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.186549 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 13:49:20.186630 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 13:49:20.186829 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":122,"thread_start_us":434,"threads_started":1}
I20260927 13:49:20.187294 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.180735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:43113 (local address 127.20.133.65:33285)
0927 13:49:20.181494 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:20.181513 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:20.181652 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:20.181978 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:20.181982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:20.182006 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:20.182450 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.182460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.183586 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.183591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.185243 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.185256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.186407 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.186414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.186526 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.186556 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:49:20.186634 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 13:49:20.186833 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":61,"thread_start_us":380,"threads_started":1}
W20260927 13:49:20.187968 22623 tablet.cc:2369] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:49:20.188441 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f753af3be948089fbf7ba071ba7aed" candidate_uuid: "d8f43c3d9d104b32aa515e06d8130c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789c301090a244dfa089fd5861180ab5" is_pre_election: true
I20260927 13:49:20.188898 22433 raft_consensus.cc:2454] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f43c3d9d104b32aa515e06d8130c69 in term 0.
I20260927 13:49:20.189540 22224 leader_election.cc:302] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 789c301090a244dfa089fd5861180ab5, d8f43c3d9d104b32aa515e06d8130c69; no voters:
I20260927 13:49:20.189632 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f753af3be948089fbf7ba071ba7aed" candidate_uuid: "d8f43c3d9d104b32aa515e06d8130c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c2accdaddd43579aba2260481a9c0b" is_pre_election: true
I20260927 13:49:20.189968 22647 raft_consensus.cc:2781] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:49:20.190102 22647 raft_consensus.cc:482] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:49:20.190105 22572 raft_consensus.cc:2454] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f43c3d9d104b32aa515e06d8130c69 in term 0.
I20260927 13:49:20.190200 22647 raft_consensus.cc:3037] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:20.192380 22647 raft_consensus.cc:504] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.192962 22647 leader_election.cc:288] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [CANDIDATE]: Term 1 election: Requested vote from peers 789c301090a244dfa089fd5861180ab5 (127.20.133.66:38119), 08c2accdaddd43579aba2260481a9c0b (127.20.133.67:43113)
I20260927 13:49:20.193519 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f753af3be948089fbf7ba071ba7aed" candidate_uuid: "d8f43c3d9d104b32aa515e06d8130c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c2accdaddd43579aba2260481a9c0b"
I20260927 13:49:20.193600 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f753af3be948089fbf7ba071ba7aed" candidate_uuid: "d8f43c3d9d104b32aa515e06d8130c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789c301090a244dfa089fd5861180ab5"
I20260927 13:49:20.193691 22572 raft_consensus.cc:3037] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:20.193845 22433 raft_consensus.cc:3037] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:20.195851 22572 raft_consensus.cc:2454] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f43c3d9d104b32aa515e06d8130c69 in term 1.
I20260927 13:49:20.196164 22433 raft_consensus.cc:2454] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f43c3d9d104b32aa515e06d8130c69 in term 1.
I20260927 13:49:20.196308 22226 leader_election.cc:302] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c2accdaddd43579aba2260481a9c0b, d8f43c3d9d104b32aa515e06d8130c69; no voters:
I20260927 13:49:20.196671 22647 raft_consensus.cc:2781] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:20.197090 22647 raft_consensus.cc:686] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 1 LEADER]: Becoming Leader. State: Replica: d8f43c3d9d104b32aa515e06d8130c69, State: Running, Role: LEADER
I20260927 13:49:20.197652 22647 consensus_queue.cc:237] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.202235 22137 catalog_manager.cc:5495] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8f43c3d9d104b32aa515e06d8130c69 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "d8f43c3d9d104b32aa515e06d8130c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } health_report { overall_health: UNKNOWN } } }
I20260927 13:49:20.229939 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.224603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.20.133.65:33077)
0927 13:49:20.224808 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.224814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.224991 (+ 177us) server_negotiation.cc:407] Connection header received
0927 13:49:20.225248 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.225252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.225339 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.225457 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:20.225855 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.226532 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.227844 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.228246 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.228372 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.228824 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:20.229381 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:20.229479 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.229641 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.229714 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:49:20.237675 22572 raft_consensus.cc:1261] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b [term 1 FOLLOWER]: Refusing update from remote peer d8f43c3d9d104b32aa515e06d8130c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:20.237670 22433 raft_consensus.cc:1261] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5 [term 1 FOLLOWER]: Refusing update from remote peer d8f43c3d9d104b32aa515e06d8130c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:20.238557 22652 consensus_queue.cc:1035] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:20.238924 22647 consensus_queue.cc:1035] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:20.258633 22657 mvcc.cc:205] Tried to move back new op lower bound from 7333957469124472832 to 7333957468973846528. Current Snapshot: MvccSnapshot[applied={T|T < 7333957469124472832}]
I20260927 13:49:20.279798 22137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:20.280048 22137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:20.284713 22137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe: Sending DeleteTablet for 3 replicas of tablet 34f753af3be948089fbf7ba071ba7aed
I20260927 13:49:20.286213 22413 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f753af3be948089fbf7ba071ba7aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:20 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:59804
I20260927 13:49:20.286244 22552 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f753af3be948089fbf7ba071ba7aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:20 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:53548
I20260927 13:49:20.287997 22274 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f753af3be948089fbf7ba071ba7aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:20 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:43200
I20260927 13:49:20.295360 22667 tablet_replica.cc:330] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: stopping tablet replica
I20260927 13:49:20.295336 22137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 13:49:20.295872 22667 raft_consensus.cc:2229] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 1 LEADER]: Raft consensus shutting down.
W20260927 13:49:20.296168 22137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:49:20.296613 22667 raft_consensus.cc:2258] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:20.299679 22666 tablet_replica.cc:330] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: stopping tablet replica
I20260927 13:49:20.300185 22666 raft_consensus.cc:2229] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:20.301017 22667 ts_tablet_manager.cc:1905] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:20.303709 22666 raft_consensus.cc:2258] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:20.304590 22665 tablet_replica.cc:330] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: stopping tablet replica
I20260927 13:49:20.307797 22274 tablet_service.cc:1467] Processing CreateTablet for tablet 1a50759fcd8c4fb9ad5a4e7e9592746c (DEFAULT_TABLE table=test-table [id=91c63235dd894007a2cce082846ba0a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:20.308282 22274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a50759fcd8c4fb9ad5a4e7e9592746c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:20.308979 22667 ts_tablet_manager.cc:1918] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:49:20.309165 22667 log.cc:1196] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/34f753af3be948089fbf7ba071ba7aed
I20260927 13:49:20.309769 22667 ts_tablet_manager.cc:1939] T 34f753af3be948089fbf7ba071ba7aed P d8f43c3d9d104b32aa515e06d8130c69: Deleting consensus metadata
I20260927 13:49:20.311355 22666 ts_tablet_manager.cc:1905] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:20.312060 22412 tablet_service.cc:1467] Processing CreateTablet for tablet 1a50759fcd8c4fb9ad5a4e7e9592746c (DEFAULT_TABLE table=test-table [id=91c63235dd894007a2cce082846ba0a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:20.312669 22412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a50759fcd8c4fb9ad5a4e7e9592746c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:20.317391 22121 catalog_manager.cc:4841] TS d8f43c3d9d104b32aa515e06d8130c69 (127.20.133.65:33077): tablet 34f753af3be948089fbf7ba071ba7aed (table test-table [id=b79f62c3b63d4cebb81407ff19f6e1bf]) successfully deleted
I20260927 13:49:20.321820 22666 ts_tablet_manager.cc:1918] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:49:20.322002 22666 log.cc:1196] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/34f753af3be948089fbf7ba071ba7aed
I20260927 13:49:20.322427 22666 ts_tablet_manager.cc:1939] T 34f753af3be948089fbf7ba071ba7aed P 08c2accdaddd43579aba2260481a9c0b: Deleting consensus metadata
I20260927 13:49:20.324014 22123 catalog_manager.cc:4841] TS 08c2accdaddd43579aba2260481a9c0b (127.20.133.67:43113): tablet 34f753af3be948089fbf7ba071ba7aed (table test-table [id=b79f62c3b63d4cebb81407ff19f6e1bf]) successfully deleted
I20260927 13:49:20.324261 22641 tablet_bootstrap.cc:492] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5: Bootstrap starting.
I20260927 13:49:20.324548 22552 tablet_service.cc:1467] Processing CreateTablet for tablet 1a50759fcd8c4fb9ad5a4e7e9592746c (DEFAULT_TABLE table=test-table [id=91c63235dd894007a2cce082846ba0a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:20.324999 22552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a50759fcd8c4fb9ad5a4e7e9592746c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:20.325953 22641 tablet_bootstrap.cc:654] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:20.327850 22641 tablet_bootstrap.cc:492] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5: No bootstrap required, opened a new log
I20260927 13:49:20.327991 22641 ts_tablet_manager.cc:1397] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 13:49:20.328680 22641 raft_consensus.cc:348] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.328860 22641 raft_consensus.cc:374] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:20.328948 22641 raft_consensus.cc:729] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789c301090a244dfa089fd5861180ab5, State: Initialized, Role: FOLLOWER
I20260927 13:49:20.329212 22641 consensus_queue.cc:260] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.331291 22643 tablet_bootstrap.cc:492] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b: Bootstrap starting.
I20260927 13:49:20.331413 22665 raft_consensus.cc:2229] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:20.331924 22665 raft_consensus.cc:2258] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:20.332031 22641 ts_tablet_manager.cc:1428] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 13:49:20.332963 22643 tablet_bootstrap.cc:654] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:20.334107 22642 tablet_bootstrap.cc:492] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69: Bootstrap starting.
I20260927 13:49:20.335835 22642 tablet_bootstrap.cc:654] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:20.338303 22642 tablet_bootstrap.cc:492] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69: No bootstrap required, opened a new log
I20260927 13:49:20.338480 22642 ts_tablet_manager.cc:1397] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260927 13:49:20.339352 22642 raft_consensus.cc:348] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.339605 22642 raft_consensus.cc:374] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:20.339715 22642 raft_consensus.cc:729] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f43c3d9d104b32aa515e06d8130c69, State: Initialized, Role: FOLLOWER
I20260927 13:49:20.340014 22642 consensus_queue.cc:260] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.340703 22642 ts_tablet_manager.cc:1428] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 13:49:20.341980 22665 ts_tablet_manager.cc:1905] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:20.343870 22643 tablet_bootstrap.cc:492] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b: No bootstrap required, opened a new log
I20260927 13:49:20.344053 22643 ts_tablet_manager.cc:1397] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260927 13:49:20.345037 22643 raft_consensus.cc:348] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.345300 22643 raft_consensus.cc:374] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:20.345407 22643 raft_consensus.cc:729] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c2accdaddd43579aba2260481a9c0b, State: Initialized, Role: FOLLOWER
I20260927 13:49:20.345748 22643 consensus_queue.cc:260] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.346627 22648 raft_consensus.cc:482] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:49:20.347082 22643 ts_tablet_manager.cc:1428] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 13:49:20.346828 22648 raft_consensus.cc:504] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.349856 22648 leader_election.cc:288] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8f43c3d9d104b32aa515e06d8130c69 (127.20.133.65:33077), 08c2accdaddd43579aba2260481a9c0b (127.20.133.67:43113)
I20260927 13:49:20.350236 22665 ts_tablet_manager.cc:1918] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:49:20.350414 22665 log.cc:1196] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/34f753af3be948089fbf7ba071ba7aed
I20260927 13:49:20.351019 22665 ts_tablet_manager.cc:1939] T 34f753af3be948089fbf7ba071ba7aed P 789c301090a244dfa089fd5861180ab5: Deleting consensus metadata
I20260927 13:49:20.361688 22121 catalog_manager.cc:4841] TS 789c301090a244dfa089fd5861180ab5 (127.20.133.66:38119): tablet 34f753af3be948089fbf7ba071ba7aed (table test-table [id=b79f62c3b63d4cebb81407ff19f6e1bf]) successfully deleted
I20260927 13:49:20.367412 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.350572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33077 (local address 127.20.133.66:57425)
0927 13:49:20.360709 (+ 10137us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:20.360729 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:20.360877 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:20.361417 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:20.361422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:20.361455 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:20.362039 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.362050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.363516 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.363522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.365221 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.365236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.366473 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.366480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.366676 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.366711 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:49:20.366852 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 13:49:20.366998 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10008,"mutex_wait_us":1,"thread_start_us":10391,"threads_started":1}
I20260927 13:49:20.367451 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.350991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:57425 (local address 127.20.133.65:33077)
0927 13:49:20.351193 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.351199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.360838 (+ 9639us) server_negotiation.cc:407] Connection header received
0927 13:49:20.361030 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.361034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.361135 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.361224 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:20.362202 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.363380 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.365406 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.366248 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.366403 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.366484 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.366928 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.367104 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:49:20.368702 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.352392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:42057 (local address 127.20.133.67:43113)
0927 13:49:20.352559 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.352565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.361907 (+ 9342us) server_negotiation.cc:407] Connection header received
0927 13:49:20.362179 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.362184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.362288 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.362344 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:20.362956 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.363990 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.366619 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.367743 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.367885 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.367959 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.368266 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.368437 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:49:20.368616 22294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a50759fcd8c4fb9ad5a4e7e9592746c" candidate_uuid: "789c301090a244dfa089fd5861180ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f43c3d9d104b32aa515e06d8130c69" is_pre_election: true
I20260927 13:49:20.369155 22294 raft_consensus.cc:2454] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789c301090a244dfa089fd5861180ab5 in term 0.
I20260927 13:49:20.369349 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.351714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:43113 (local address 127.20.133.66:42057)
0927 13:49:20.361782 (+ 10068us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:20.361804 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:20.362012 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:20.362499 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:20.362502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:20.362520 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:20.362800 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.362808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.364138 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.364144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.366418 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.366437 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.367934 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.367941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.368092 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.368116 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 13:49:20.368891 (+ 775us) client_negotiation.cc:240] Negotiation successful
0927 13:49:20.369038 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9820,"thread_start_us":9605,"threads_started":1}
I20260927 13:49:20.370061 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a50759fcd8c4fb9ad5a4e7e9592746c" candidate_uuid: "789c301090a244dfa089fd5861180ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c2accdaddd43579aba2260481a9c0b" is_pre_election: true
I20260927 13:49:20.370085 22364 leader_election.cc:302] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 789c301090a244dfa089fd5861180ab5, d8f43c3d9d104b32aa515e06d8130c69; no voters:
I20260927 13:49:20.370317 22572 raft_consensus.cc:2454] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789c301090a244dfa089fd5861180ab5 in term 0.
I20260927 13:49:20.370558 22648 raft_consensus.cc:2781] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:49:20.370671 22648 raft_consensus.cc:482] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:49:20.370766 22648 raft_consensus.cc:3037] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:20.373104 22648 raft_consensus.cc:504] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.373736 22648 leader_election.cc:288] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [CANDIDATE]: Term 1 election: Requested vote from peers d8f43c3d9d104b32aa515e06d8130c69 (127.20.133.65:33077), 08c2accdaddd43579aba2260481a9c0b (127.20.133.67:43113)
I20260927 13:49:20.374404 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a50759fcd8c4fb9ad5a4e7e9592746c" candidate_uuid: "789c301090a244dfa089fd5861180ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c2accdaddd43579aba2260481a9c0b"
I20260927 13:49:20.374662 22572 raft_consensus.cc:3037] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:20.377864 22572 raft_consensus.cc:2454] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789c301090a244dfa089fd5861180ab5 in term 1.
I20260927 13:49:20.378334 22294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a50759fcd8c4fb9ad5a4e7e9592746c" candidate_uuid: "789c301090a244dfa089fd5861180ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f43c3d9d104b32aa515e06d8130c69"
I20260927 13:49:20.378414 22366 leader_election.cc:302] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c2accdaddd43579aba2260481a9c0b, 789c301090a244dfa089fd5861180ab5; no voters:
I20260927 13:49:20.378597 22294 raft_consensus.cc:3037] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:20.378762 22648 raft_consensus.cc:2781] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:20.379244 22648 raft_consensus.cc:686] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 1 LEADER]: Becoming Leader. State: Replica: 789c301090a244dfa089fd5861180ab5, State: Running, Role: LEADER
I20260927 13:49:20.379940 22648 consensus_queue.cc:237] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.381961 22294 raft_consensus.cc:2454] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789c301090a244dfa089fd5861180ab5 in term 1.
I20260927 13:49:20.383876 22137 catalog_manager.cc:5495] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 789c301090a244dfa089fd5861180ab5 (127.20.133.66). New cstate: current_term: 1 leader_uuid: "789c301090a244dfa089fd5861180ab5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } health_report { overall_health: UNKNOWN } } }
Sep 27 13:49:20 dist-test-slave-q25r krb5kdc[22092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:20.437129 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.423095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.20.133.126:34807)
0927 13:49:20.423359 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.423366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.423935 (+ 569us) server_negotiation.cc:407] Connection header received
0927 13:49:20.424090 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.424096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.424203 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.424328 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:20.425677 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.426542 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.427827 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.428267 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.428414 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.431881 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:20.431886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:20.431890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:20.431921 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:20.434454 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.435149 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.435152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.435155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.435283 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.435611 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.435615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.435617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.435907 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:20.436034 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.436481 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.436813 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":75}
I20260927 13:49:20.437136 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.423126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34807 (local address 127.0.0.1:42648)
0927 13:49:20.423822 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:20.423840 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:20.423978 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:20.424414 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:20.424418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:20.425111 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:20.425520 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.425534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.426684 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.426689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.427672 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.427684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.428483 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.428493 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.428648 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.429308 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:20.429326 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:20.431682 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:20.434594 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:20.434610 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:20.434619 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:20.435034 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:20.435386 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:20.435391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:20.435395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:20.435515 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:20.436030 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:20.436037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:20.436203 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 13:49:20.436374 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 13:49:20.436672 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":383,"threads_started":1}
I20260927 13:49:20.442653 22137 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:42648:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 13:49:20.443746 22137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:49:20.451263 22412 tablet_service.cc:1467] Processing CreateTablet for tablet 2394f92b60fe4ac9ae975e001686f3f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ef9efc9148754ca58422ceda1eaaf31c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:49:20.451226 22274 tablet_service.cc:1467] Processing CreateTablet for tablet 2394f92b60fe4ac9ae975e001686f3f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ef9efc9148754ca58422ceda1eaaf31c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:49:20.451812 22412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2394f92b60fe4ac9ae975e001686f3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:20.451845 22274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2394f92b60fe4ac9ae975e001686f3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:20.457412 22552 tablet_service.cc:1467] Processing CreateTablet for tablet 2394f92b60fe4ac9ae975e001686f3f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ef9efc9148754ca58422ceda1eaaf31c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:49:20.458019 22552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2394f92b60fe4ac9ae975e001686f3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:20.458701 22641 tablet_bootstrap.cc:492] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5: Bootstrap starting.
I20260927 13:49:20.459317 22642 tablet_bootstrap.cc:492] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: Bootstrap starting.
I20260927 13:49:20.460939 22641 tablet_bootstrap.cc:654] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:20.461102 22642 tablet_bootstrap.cc:654] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:20.465601 22643 tablet_bootstrap.cc:492] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b: Bootstrap starting.
I20260927 13:49:20.467576 22643 tablet_bootstrap.cc:654] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:20.477735 22641 tablet_bootstrap.cc:492] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5: No bootstrap required, opened a new log
I20260927 13:49:20.477936 22641 ts_tablet_manager.cc:1397] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.012s
I20260927 13:49:20.478118 22643 tablet_bootstrap.cc:492] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b: No bootstrap required, opened a new log
I20260927 13:49:20.478304 22643 ts_tablet_manager.cc:1397] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260927 13:49:20.478737 22641 raft_consensus.cc:348] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.478992 22641 raft_consensus.cc:374] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:20.479069 22641 raft_consensus.cc:729] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789c301090a244dfa089fd5861180ab5, State: Initialized, Role: FOLLOWER
I20260927 13:49:20.479048 22643 raft_consensus.cc:348] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.479301 22643 raft_consensus.cc:374] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:20.479382 22643 raft_consensus.cc:729] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c2accdaddd43579aba2260481a9c0b, State: Initialized, Role: FOLLOWER
I20260927 13:49:20.479362 22641 consensus_queue.cc:260] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.479796 22648 tablet_replica.cc:439] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } } }
I20260927 13:49:20.480067 22648 tablet_replica.cc:442] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:20.480307 22642 tablet_bootstrap.cc:492] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: No bootstrap required, opened a new log
I20260927 13:49:20.480530 22642 ts_tablet_manager.cc:1397] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.009s
I20260927 13:49:20.481297 22642 raft_consensus.cc:348] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.481544 22642 raft_consensus.cc:374] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:20.481634 22642 raft_consensus.cc:729] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f43c3d9d104b32aa515e06d8130c69, State: Initialized, Role: FOLLOWER
I20260927 13:49:20.481917 22642 consensus_queue.cc:260] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.482364 22647 tablet_replica.cc:439] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } } }
I20260927 13:49:20.482632 22647 tablet_replica.cc:442] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:20.483484 22641 ts_tablet_manager.cc:1428] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260927 13:49:20.483745 22642 ts_tablet_manager.cc:1428] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 13:49:20.484105 22643 consensus_queue.cc:260] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.484565 22649 tablet_replica.cc:439] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } } }
I20260927 13:49:20.484869 22649 tablet_replica.cc:442] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:20.485258 22643 ts_tablet_manager.cc:1428] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260927 13:49:20.535290 22647 raft_consensus.cc:482] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:49:20.535475 22647 raft_consensus.cc:504] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.536165 22647 leader_election.cc:288] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789c301090a244dfa089fd5861180ab5 (127.20.133.66:38119), 08c2accdaddd43579aba2260481a9c0b (127.20.133.67:43113)
I20260927 13:49:20.536757 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2394f92b60fe4ac9ae975e001686f3f6" candidate_uuid: "d8f43c3d9d104b32aa515e06d8130c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789c301090a244dfa089fd5861180ab5" is_pre_election: true
I20260927 13:49:20.536827 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2394f92b60fe4ac9ae975e001686f3f6" candidate_uuid: "d8f43c3d9d104b32aa515e06d8130c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c2accdaddd43579aba2260481a9c0b" is_pre_election: true
I20260927 13:49:20.537032 22433 raft_consensus.cc:2454] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f43c3d9d104b32aa515e06d8130c69 in term 0.
I20260927 13:49:20.537065 22572 raft_consensus.cc:2454] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f43c3d9d104b32aa515e06d8130c69 in term 0.
I20260927 13:49:20.537546 22224 leader_election.cc:302] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08c2accdaddd43579aba2260481a9c0b, 789c301090a244dfa089fd5861180ab5, d8f43c3d9d104b32aa515e06d8130c69; no voters:
I20260927 13:49:20.537887 22647 raft_consensus.cc:2781] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:49:20.538014 22647 raft_consensus.cc:482] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:49:20.538116 22647 raft_consensus.cc:3037] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:20.540603 22647 raft_consensus.cc:504] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.541183 22647 leader_election.cc:288] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [CANDIDATE]: Term 1 election: Requested vote from peers 789c301090a244dfa089fd5861180ab5 (127.20.133.66:38119), 08c2accdaddd43579aba2260481a9c0b (127.20.133.67:43113)
I20260927 13:49:20.541683 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2394f92b60fe4ac9ae975e001686f3f6" candidate_uuid: "d8f43c3d9d104b32aa515e06d8130c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c2accdaddd43579aba2260481a9c0b"
I20260927 13:49:20.541767 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2394f92b60fe4ac9ae975e001686f3f6" candidate_uuid: "d8f43c3d9d104b32aa515e06d8130c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789c301090a244dfa089fd5861180ab5"
I20260927 13:49:20.541898 22572 raft_consensus.cc:3037] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:20.542016 22433 raft_consensus.cc:3037] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:20.544005 22572 raft_consensus.cc:2454] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f43c3d9d104b32aa515e06d8130c69 in term 1.
I20260927 13:49:20.544445 22226 leader_election.cc:302] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c2accdaddd43579aba2260481a9c0b, d8f43c3d9d104b32aa515e06d8130c69; no voters:
I20260927 13:49:20.544713 22647 raft_consensus.cc:2781] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:20.544857 22647 raft_consensus.cc:686] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [term 1 LEADER]: Becoming Leader. State: Replica: d8f43c3d9d104b32aa515e06d8130c69, State: Running, Role: LEADER
I20260927 13:49:20.545121 22433 raft_consensus.cc:2454] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f43c3d9d104b32aa515e06d8130c69 in term 1.
I20260927 13:49:20.545117 22647 consensus_queue.cc:237] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } }
I20260927 13:49:20.546851 22662 tablet_replica.cc:439] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: TxnStatusTablet state changed. Reason: New leader d8f43c3d9d104b32aa515e06d8130c69. Latest consensus state: current_term: 1 leader_uuid: "d8f43c3d9d104b32aa515e06d8130c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } } }
I20260927 13:49:20.547281 22662 tablet_replica.cc:442] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:49:20.548161 22685 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 13:49:20.548436 22685 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 13:49:20.549438 22136 catalog_manager.cc:5495] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8f43c3d9d104b32aa515e06d8130c69 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "d8f43c3d9d104b32aa515e06d8130c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } health_report { overall_health: UNKNOWN } } }
I20260927 13:49:20.571447 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.565995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.20.133.65:33077)
0927 13:49:20.566173 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.566177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.566195 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:49:20.566401 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.566404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.566471 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.566549 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:20.566993 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.567867 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.569726 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.570164 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.570272 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.570798 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:20.570916 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:20.570981 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.571146 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.571210 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:49:20.571612 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.565782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33077 (local address 127.0.0.1:37014)
0927 13:49:20.566048 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:20.566071 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:20.566220 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:20.566613 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:20.566618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:20.566633 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:49:20.566817 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.566828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.568025 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.568032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.569567 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.569595 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.570331 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.570338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.570546 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.570639 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:49:20.571015 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:49:20.571046 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 13:49:20.571135 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 13:49:20.571297 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 13:49:20.575387 22433 raft_consensus.cc:1261] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5 [term 1 FOLLOWER]: Refusing update from remote peer d8f43c3d9d104b32aa515e06d8130c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:20.575423 22572 raft_consensus.cc:1261] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b [term 1 FOLLOWER]: Refusing update from remote peer d8f43c3d9d104b32aa515e06d8130c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:20.576061 22647 consensus_queue.cc:1035] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:20.576361 22662 consensus_queue.cc:1035] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:20.579370 22648 tablet_replica.cc:439] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5: TxnStatusTablet state changed. Reason: New leader d8f43c3d9d104b32aa515e06d8130c69. Latest consensus state: current_term: 1 leader_uuid: "d8f43c3d9d104b32aa515e06d8130c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } } }
I20260927 13:49:20.579658 22648 tablet_replica.cc:442] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:20.579660 22649 tablet_replica.cc:439] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b: TxnStatusTablet state changed. Reason: New leader d8f43c3d9d104b32aa515e06d8130c69. Latest consensus state: current_term: 1 leader_uuid: "d8f43c3d9d104b32aa515e06d8130c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } } }
I20260927 13:49:20.579903 22649 tablet_replica.cc:442] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:20.581897 22655 mvcc.cc:205] Tried to move back new op lower bound from 7333957470510313472 to 7333957470395793408. Current Snapshot: MvccSnapshot[applied={T|T < 7333957470510313472}]
I20260927 13:49:20.582633 22656 mvcc.cc:205] Tried to move back new op lower bound from 7333957470510313472 to 7333957470395793408. Current Snapshot: MvccSnapshot[applied={T|T < 7333957470510313472}]
I20260927 13:49:20.583197 22648 tablet_replica.cc:439] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8f43c3d9d104b32aa515e06d8130c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } } }
I20260927 13:49:20.583485 22648 tablet_replica.cc:442] T 2394f92b60fe4ac9ae975e001686f3f6 P 789c301090a244dfa089fd5861180ab5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:20.583477 22652 tablet_replica.cc:439] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8f43c3d9d104b32aa515e06d8130c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } } }
I20260927 13:49:20.583823 22652 tablet_replica.cc:442] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:49:20.584055 22649 tablet_replica.cc:439] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8f43c3d9d104b32aa515e06d8130c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } } }
I20260927 13:49:20.584272 22649 tablet_replica.cc:442] T 2394f92b60fe4ac9ae975e001686f3f6 P 08c2accdaddd43579aba2260481a9c0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:20.585628 22662 tablet_replica.cc:439] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8f43c3d9d104b32aa515e06d8130c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 } } peers { permanent_uuid: "789c301090a244dfa089fd5861180ab5" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38119 } } peers { permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 } } }
I20260927 13:49:20.585868 22662 tablet_replica.cc:442] T 2394f92b60fe4ac9ae975e001686f3f6 P d8f43c3d9d104b32aa515e06d8130c69: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:49:20.596405 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.590478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.20.133.66:38119)
0927 13:49:20.590654 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.590660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.590680 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:49:20.590892 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.590897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.591009 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.591159 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:20.591529 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.592271 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.594162 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.594576 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.594669 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.595163 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:20.595747 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:20.595848 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.596014 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.596100 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:49:20.613191 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.607356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:43461 (local address 127.20.133.65:33077)
0927 13:49:20.607554 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.607558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.608112 (+ 554us) server_negotiation.cc:407] Connection header received
0927 13:49:20.608307 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.608313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.608427 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.608507 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:20.609017 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.609808 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.611230 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.611711 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.611838 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.612365 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:20.612518 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:20.612598 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.612749 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.612834 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:49:20.613241 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.607381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33077 (local address 127.20.133.66:43461)
0927 13:49:20.608001 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:20.608017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:20.608167 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:20.608537 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:20.608541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:20.608554 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:49:20.608854 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.608864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.609955 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.609959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.611079 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.611093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.611924 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.611930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.612097 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.612162 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:49:20.612614 (+ 452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:49:20.612626 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 13:49:20.612699 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 13:49:20.612855 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":339,"threads_started":1}
I20260927 13:49:20.624236 22572 raft_consensus.cc:1261] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b [term 1 FOLLOWER]: Refusing update from remote peer 789c301090a244dfa089fd5861180ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:20.624727 22294 raft_consensus.cc:1261] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69 [term 1 FOLLOWER]: Refusing update from remote peer 789c301090a244dfa089fd5861180ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:20.624850 22648 consensus_queue.cc:1035] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08c2accdaddd43579aba2260481a9c0b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:20.625524 22648 consensus_queue.cc:1035] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8f43c3d9d104b32aa515e06d8130c69" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:20.662894 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.656221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:56503 (local address 127.20.133.66:38119)
0927 13:49:20.656471 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.656477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.657038 (+ 561us) server_negotiation.cc:407] Connection header received
0927 13:49:20.657254 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.657259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.657345 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.657427 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:20.658291 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.659304 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.660912 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.661450 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.661573 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.662131 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:20.662269 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:20.662347 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.662521 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.662612 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:49:20.663229 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.656118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:38119 (local address 127.20.133.65:56503)
0927 13:49:20.656925 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:20.656946 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:20.657102 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:20.657465 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:20.657470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:20.657487 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:49:20.658108 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.658122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.659451 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.659456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.660768 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:20.660783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.661654 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.661660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:20.661847 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.661932 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:49:20.662413 (+ 481us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:49:20.662425 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 13:49:20.662501 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 13:49:20.662807 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":403,"threads_started":1}
I20260927 13:49:20.699710 22135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:20.700016 22135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:20.703980 22135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cae2f07f7ac425f98d16ee034d548fe: Sending DeleteTablet for 3 replicas of tablet 1a50759fcd8c4fb9ad5a4e7e9592746c
I20260927 13:49:20.705224 22552 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a50759fcd8c4fb9ad5a4e7e9592746c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:20 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:53548
I20260927 13:49:20.705556 22666 tablet_replica.cc:330] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b: stopping tablet replica
I20260927 13:49:20.705489 22412 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a50759fcd8c4fb9ad5a4e7e9592746c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:20 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:59804
I20260927 13:49:20.705767 22666 raft_consensus.cc:2229] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:20.705870 22665 tablet_replica.cc:330] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5: stopping tablet replica
I20260927 13:49:20.705974 22666 raft_consensus.cc:2258] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:20.706089 22665 raft_consensus.cc:2229] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:49:20.706652 22665 raft_consensus.cc:2258] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:20.707341 22666 ts_tablet_manager.cc:1905] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:20.707720 22274 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a50759fcd8c4fb9ad5a4e7e9592746c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:20 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:43200
I20260927 13:49:20.708020 22667 tablet_replica.cc:330] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69: stopping tablet replica
I20260927 13:49:20.708212 22667 raft_consensus.cc:2229] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:20.708554 22667 raft_consensus.cc:2258] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:20.709972 22667 ts_tablet_manager.cc:1905] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:20.714548 22666 ts_tablet_manager.cc:1918] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:49:20.714710 22666 log.cc:1196] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/1a50759fcd8c4fb9ad5a4e7e9592746c
I20260927 13:49:20.715549 22666 ts_tablet_manager.cc:1939] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 08c2accdaddd43579aba2260481a9c0b: Deleting consensus metadata
I20260927 13:49:20.716537 22665 ts_tablet_manager.cc:1905] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:20.716885 22123 catalog_manager.cc:4841] TS 08c2accdaddd43579aba2260481a9c0b (127.20.133.67:43113): tablet 1a50759fcd8c4fb9ad5a4e7e9592746c (table test-table [id=91c63235dd894007a2cce082846ba0a8]) successfully deleted
I20260927 13:49:20.724505 22665 ts_tablet_manager.cc:1918] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:49:20.724694 22667 ts_tablet_manager.cc:1918] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:49:20.724895 22667 log.cc:1196] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/1a50759fcd8c4fb9ad5a4e7e9592746c
I20260927 13:49:20.724895 22665 log.cc:1196] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/1a50759fcd8c4fb9ad5a4e7e9592746c
I20260927 13:49:20.725387 22667 ts_tablet_manager.cc:1939] T 1a50759fcd8c4fb9ad5a4e7e9592746c P d8f43c3d9d104b32aa515e06d8130c69: Deleting consensus metadata
I20260927 13:49:20.725548 22665 ts_tablet_manager.cc:1939] T 1a50759fcd8c4fb9ad5a4e7e9592746c P 789c301090a244dfa089fd5861180ab5: Deleting consensus metadata
I20260927 13:49:20.726711 22121 catalog_manager.cc:4841] TS d8f43c3d9d104b32aa515e06d8130c69 (127.20.133.65:33077): tablet 1a50759fcd8c4fb9ad5a4e7e9592746c (table test-table [id=91c63235dd894007a2cce082846ba0a8]) successfully deleted
I20260927 13:49:20.727015 22121 catalog_manager.cc:4841] TS 789c301090a244dfa089fd5861180ab5 (127.20.133.66:38119): tablet 1a50759fcd8c4fb9ad5a4e7e9592746c (table test-table [id=91c63235dd894007a2cce082846ba0a8]) successfully deleted
Sep 27 13:49:20 dist-test-slave-q25r krb5kdc[22092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516959, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:20.748689 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.728143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.20.133.65:33077)
0927 13:49:20.728378 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.728386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.729103 (+ 717us) server_negotiation.cc:407] Connection header received
0927 13:49:20.729287 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.729293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.729389 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.729499 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:20.731399 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.732648 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.734606 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.735283 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.735434 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.740790 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:20.740813 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:20.740837 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:20.740892 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:20.745196 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.746136 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.746146 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.746150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.746323 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.746782 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.746787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.746791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.747275 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:20.747609 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.747925 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.748221 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 13:49:20.749500 22339 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37030
I20260927 13:49:20.770510 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.753097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.20.133.126:34807)
0927 13:49:20.753341 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.753347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.753761 (+ 414us) server_negotiation.cc:407] Connection header received
0927 13:49:20.754025 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.754031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.754119 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.754192 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:20.756133 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.757392 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.759231 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.759901 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.760029 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.763259 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:20.763266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:20.763271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:20.763362 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:20.767128 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.768132 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.768139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.768143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.768323 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.768808 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.768813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.768816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.769219 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:20.769534 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.769863 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.770124 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260927 13:49:20.771426 22135 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35644
I20260927 13:49:20.774210 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22208
I20260927 13:49:20.798018 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.775716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.20.133.65:33077)
0927 13:49:20.775921 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.775927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.781019 (+ 5092us) server_negotiation.cc:407] Connection header received
0927 13:49:20.781181 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.781187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.781289 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.781422 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:20.783081 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.784245 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.786266 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.787013 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.787156 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.790652 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:20.790660 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:20.790665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:20.790718 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:20.794523 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.795493 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.795500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.795503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.795695 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.796101 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.796106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.796109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.796510 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:20.796798 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.797091 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.797354 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 13:49:20.798832 22339 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37046
W20260927 13:49:20.799479 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22208: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:49:20.799679 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 22208
I20260927 13:49:20.818770 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22348
Sep 27 13:49:20 dist-test-slave-q25r krb5kdc[22092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516959, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:49:20.836300 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.819545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.20.133.66:38119)
0927 13:49:20.819724 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.819730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.819770 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:49:20.819986 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.819991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.820076 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.820164 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:20.821435 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.822626 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.823914 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.824643 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.824802 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.828873 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:20.828892 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:20.828915 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:20.828967 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:20.832944 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.833917 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.833928 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.833932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.834117 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.834549 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.834553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.834557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.835061 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:20.835420 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.835672 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.835909 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 13:49:20.837147 22478 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51478
W20260927 13:49:20.837890 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22348: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:49:20.838068 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 22348
I20260927 13:49:20.852648 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22487
Sep 27 13:49:20 dist-test-slave-q25r krb5kdc[22092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516959, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:49:20.868832 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:20.853455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.20.133.67:43113)
0927 13:49:20.853662 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:20.853668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:20.853688 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:49:20.853816 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:20.853821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:20.853910 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:20.853997 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:20.855235 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.855966 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.857379 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:20.857896 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:20.858012 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:20.862571 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:20.862599 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:20.862615 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:20.862666 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:20.865759 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.866604 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.866612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.866615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.866785 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:20.867177 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:20.867181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:20.867183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:20.867615 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:20.867951 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:20.868243 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 13:49:20.868516 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 13:49:20.869585 22617 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41104
W20260927 13:49:20.870267 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22487: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:49:20.870404 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 22487
I20260927 13:49:20.883737 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22105
I20260927 13:49:20.885030 22177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:21.001009 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 22105
2026-09-27T13:49:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4386 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:49:21 dist-test-slave-q25r krb5kdc[22707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:49:21 dist-test-slave-q25r krb5kdc[22707](info): set up 2 sockets
Sep 27 13:49:21 dist-test-slave-q25r krb5kdc[22707](info): commencing operation
krb5kdc: starting...
W20260927 13:49:22.005137 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:49:22 dist-test-slave-q25r krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:49:22.216444 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:43085
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:43085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:22.554293 22720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:22.554646 22720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:22.554775 22720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:22.562423 22720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:22.562572 22720 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 13:49:22.562623 22720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:22.562728 22720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:22.562809 22720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:22.562913 22720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:22.573647 22720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:43085
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:43085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:22.575038 22720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:22.576474 22720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:22.590265 22726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:22.590267 22728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:22.595567 22725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:22.596825 22720 server_base.cc:1034] running on GCE node
I20260927 13:49:22.597797 22720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:22.599447 22720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:22.600768 22720 hybrid_clock.cc:648] HybridClock initialized: now 1790516962600681 us; error 51 us; skew 500 ppm
I20260927 13:49:22.601177 22720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:22 dist-test-slave-q25r krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516962, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:22.605176 22720 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:49:22.607072 22720 webserver.cc:460] Webserver started at http://127.20.133.126:34725/ using document root <none> and password file <none>
I20260927 13:49:22.607838 22720 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:22.607941 22720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:22.608235 22720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:22.611512 22720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "e59d5973ba96416b8e6c75f7bdc48468"
format_stamp: "Formatted at 2026-09-27 13:49:22 on dist-test-slave-q25r"
server_key: "2630aa9918eb0c7d65754d2a38174cff"
server_key_iv: "77ba5313fe707e10825ee2737abf32b4"
server_key_version: "encryptionkey@0"
I20260927 13:49:22.612421 22720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "e59d5973ba96416b8e6c75f7bdc48468"
format_stamp: "Formatted at 2026-09-27 13:49:22 on dist-test-slave-q25r"
server_key: "2630aa9918eb0c7d65754d2a38174cff"
server_key_iv: "77ba5313fe707e10825ee2737abf32b4"
server_key_version: "encryptionkey@0"
I20260927 13:49:22.618314 22720 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 13:49:22.622339 22735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:22.624284 22720 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 13:49:22.624737 22720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "e59d5973ba96416b8e6c75f7bdc48468"
format_stamp: "Formatted at 2026-09-27 13:49:22 on dist-test-slave-q25r"
server_key: "2630aa9918eb0c7d65754d2a38174cff"
server_key_iv: "77ba5313fe707e10825ee2737abf32b4"
server_key_version: "encryptionkey@0"
I20260927 13:49:22.625011 22720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:22.641433 22720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:22.646828 22720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:22.647158 22720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:22.671057 22720 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:43085
I20260927 13:49:22.671094 22796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:43085 every 8 connection(s)
I20260927 13:49:22.672801 22720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:22.677156 22797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:22.678963 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 22720
I20260927 13:49:22.679152 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:22.679883 21013 external_mini_cluster.cc:1428] Setting key 0c1a80b332c126574f5f6700123d66d5
I20260927 13:49:22.690629 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468: Bootstrap starting.
I20260927 13:49:22.694416 22797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:22.695508 22797 log.cc:824] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:49:22 dist-test-slave-q25r krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:22.699277 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468: No bootstrap required, opened a new log
I20260927 13:49:22.709046 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:22.682441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.20.133.126:43085)
0927 13:49:22.683465 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:22.683485 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:22.683547 (+ 62us) server_negotiation.cc:407] Connection header received
0927 13:49:22.685964 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:22.686001 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:22.686581 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:22.687098 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:22.688804 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:22.690754 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:22.692281 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:22.693382 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:22.693611 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:22.698612 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:22.698623 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:22.698647 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:22.698697 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:22.702986 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:22.703941 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:22.703952 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:22.703962 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:22.704213 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:22.704690 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:22.704696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:22.704699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:22.705554 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:22.705903 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:22.706414 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 13:49:22.707101 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":394,"threads_started":1}
I20260927 13:49:22.710641 22797 raft_consensus.cc:348] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d5973ba96416b8e6c75f7bdc48468" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43085 } }
I20260927 13:49:22.711083 22797 raft_consensus.cc:374] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:22.711200 22797 raft_consensus.cc:729] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59d5973ba96416b8e6c75f7bdc48468, State: Initialized, Role: FOLLOWER
I20260927 13:49:22.711954 22797 consensus_queue.cc:260] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d5973ba96416b8e6c75f7bdc48468" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43085 } }
I20260927 13:49:22.712262 22797 raft_consensus.cc:388] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:22.712433 22797 raft_consensus.cc:482] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:22.712623 22797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:22.714943 22797 raft_consensus.cc:504] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d5973ba96416b8e6c75f7bdc48468" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43085 } }
I20260927 13:49:22.715451 22797 leader_election.cc:302] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e59d5973ba96416b8e6c75f7bdc48468; no voters:
I20260927 13:49:22.716089 22797 leader_election.cc:288] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:22.716300 22802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:22.717708 22802 raft_consensus.cc:686] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [term 1 LEADER]: Becoming Leader. State: Replica: e59d5973ba96416b8e6c75f7bdc48468, State: Running, Role: LEADER
I20260927 13:49:22.718452 22797 sys_catalog.cc:564] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:22.718415 22802 consensus_queue.cc:237] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d5973ba96416b8e6c75f7bdc48468" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43085 } }
I20260927 13:49:22.721905 22804 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e59d5973ba96416b8e6c75f7bdc48468. Latest consensus state: current_term: 1 leader_uuid: "e59d5973ba96416b8e6c75f7bdc48468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d5973ba96416b8e6c75f7bdc48468" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43085 } } }
I20260927 13:49:22.722384 22804 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:22.721906 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e59d5973ba96416b8e6c75f7bdc48468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59d5973ba96416b8e6c75f7bdc48468" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43085 } } }
I20260927 13:49:22.723086 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:22.724581 22810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:22.731233 22810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:22.743861 22810 catalog_manager.cc:1347] Generated new cluster ID: 3b828fc8d79049c6b7581c95ef321230
I20260927 13:49:22.744006 22810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:22.753746 22810 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:22.756062 22810 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:22.764575 22810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468: Generated new TSK 0
I20260927 13:49:22.765880 22810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:49:22.811394 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:43085
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:23.145766 22823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:23.146051 22823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:23.146153 22823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:23.153594 22823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:23.153723 22823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:23.153808 22823 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 13:49:23.153860 22823 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:49:23.153928 22823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:49:23.163693 22823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.133.126:43085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:23.165217 22823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:23.166567 22823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:23.179783 22828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:23.179843 22829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:23.179801 22831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:23.180601 22823 server_base.cc:1034] running on GCE node
I20260927 13:49:23.181216 22823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:23.182168 22823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:23.183377 22823 hybrid_clock.cc:648] HybridClock initialized: now 1790516963183312 us; error 54 us; skew 500 ppm
I20260927 13:49:23.183687 22823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:23 dist-test-slave-q25r krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516963, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:23.186880 22823 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:49:23.188375 22823 webserver.cc:460] Webserver started at http://127.20.133.65:33295/ using document root <none> and password file <none>
I20260927 13:49:23.189064 22823 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:23.189179 22823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:23.189493 22823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:23.192505 22823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9"
format_stamp: "Formatted at 2026-09-27 13:49:23 on dist-test-slave-q25r"
server_key: "c0dad552f0555a76f4ba6b73c282a45d"
server_key_iv: "0e48c45ecdb464e1f3cd852c0fc2c0a7"
server_key_version: "encryptionkey@0"
I20260927 13:49:23.193269 22823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9"
format_stamp: "Formatted at 2026-09-27 13:49:23 on dist-test-slave-q25r"
server_key: "c0dad552f0555a76f4ba6b73c282a45d"
server_key_iv: "0e48c45ecdb464e1f3cd852c0fc2c0a7"
server_key_version: "encryptionkey@0"
I20260927 13:49:23.198809 22823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:49:23.202522 22838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:23.204069 22823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:49:23.204257 22823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9"
format_stamp: "Formatted at 2026-09-27 13:49:23 on dist-test-slave-q25r"
server_key: "c0dad552f0555a76f4ba6b73c282a45d"
server_key_iv: "0e48c45ecdb464e1f3cd852c0fc2c0a7"
server_key_version: "encryptionkey@0"
I20260927 13:49:23.204522 22823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:23.229466 22823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:23.233814 22823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:23.234090 22823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:23.236889 22823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:23.237004 22823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:23.237109 22823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:23.237193 22823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:49:23 dist-test-slave-q25r krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516963, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:23.257895 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:23.237545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:52279 (local address 127.20.133.126:43085)
0927 13:49:23.238025 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:23.238030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:23.239207 (+ 1177us) server_negotiation.cc:407] Connection header received
0927 13:49:23.241389 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:23.241396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:23.241483 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:23.241606 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:23.243743 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.244554 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.246151 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.246668 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.246779 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:23.251422 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:23.251428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:23.251433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:23.251477 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:23.254607 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:23.255347 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:23.255352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:23.255355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:23.255470 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:23.255914 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:23.255918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:23.255920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:23.256205 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:23.256357 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:23.257250 (+ 893us) server_negotiation.cc:299] Negotiation successful
0927 13:49:23.257479 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":258,"threads_started":1}
I20260927 13:49:23.259495 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:23.238191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43085 (local address 127.20.133.65:52279)
0927 13:49:23.239034 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:23.239072 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:23.241088 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:23.241910 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:23.241919 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:23.242633 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:23.243546 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:23.243580 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.244703 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.244709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:23.246000 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:23.246012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.246950 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.246959 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:23.247297 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:23.248121 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:23.248152 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:23.251171 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:23.254748 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:23.254758 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:23.254777 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:23.255222 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:23.255586 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:23.255591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:23.255595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:23.255803 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:23.256357 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:23.256367 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:23.256817 (+ 450us) client_negotiation.cc:769] Sending connection context
0927 13:49:23.257162 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 13:49:23.257792 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":364,"threads_started":1}
I20260927 13:49:23.283109 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:23.265968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:37081 (local address 127.20.133.126:43085)
0927 13:49:23.266158 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:23.266163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:23.268986 (+ 2823us) server_negotiation.cc:407] Connection header received
0927 13:49:23.269212 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:23.269219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:23.269316 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:23.269431 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:23.270761 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.271706 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.274224 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.274910 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.275049 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:23.277557 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:23.277563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:23.277566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:23.277608 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:23.280157 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:23.281064 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:23.281071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:23.281075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:23.281278 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:23.281790 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:23.281795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:23.281799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:23.282075 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:23.282198 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:23.282678 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 13:49:23.282873 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:49:23.283548 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:23.266068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43085 (local address 127.20.133.65:37081)
0927 13:49:23.268841 (+ 2773us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:23.268861 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:23.269072 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:23.269508 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:23.269514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:23.270165 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:23.270572 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:23.270586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.271893 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.271900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:23.273960 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:23.273985 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.275184 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.275191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:23.275319 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:23.276045 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:23.276065 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:23.277246 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:23.280321 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:23.280326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:23.280363 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:23.280884 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:23.281418 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:23.281424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:23.281428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:23.281634 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:23.282188 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:23.282194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:23.282425 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 13:49:23.282595 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 13:49:23.282999 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2623,"thread_start_us":2463,"threads_started":1}
I20260927 13:49:23.287282 22823 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:41361
I20260927 13:49:23.287339 22958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:41361 every 8 connection(s)
I20260927 13:49:23.288658 22823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:49:23.290613 22959 heartbeater.cc:347] Connected to a master server at 127.20.133.126:43085
I20260927 13:49:23.290840 22959 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:23.291555 22959 heartbeater.cc:511] Master 127.20.133.126:43085 requested a full tablet report, sending...
I20260927 13:49:23.291788 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 22823
I20260927 13:49:23.291986 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:49:23.292752 21013 external_mini_cluster.cc:1428] Setting key eaf0ff78da7f705cde904159e8a88e77
I20260927 13:49:23.293896 22752 ts_manager.cc:194] Registered new tserver with Master: f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65:41361)
I20260927 13:49:23.295735 22752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:52279
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:49:23.335410 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:43085
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:23.653457 22963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:23.653805 22963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:23.653971 22963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:23.661592 22963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:23.661746 22963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:23.661868 22963 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 13:49:23.661942 22963 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:49:23.662057 22963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:49:23.672358 22963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.133.126:43085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:23.673614 22963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:23.675022 22963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:23.688817 22968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:23.688745 22969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:23.688771 22971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:23.689592 22963 server_base.cc:1034] running on GCE node
I20260927 13:49:23.690171 22963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:23.691135 22963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:23.692379 22963 hybrid_clock.cc:648] HybridClock initialized: now 1790516963692295 us; error 40 us; skew 500 ppm
I20260927 13:49:23.692696 22963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:23 dist-test-slave-q25r krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516963, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:23.696305 22963 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:49:23.698081 22963 webserver.cc:460] Webserver started at http://127.20.133.66:36981/ using document root <none> and password file <none>
I20260927 13:49:23.698783 22963 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:23.698907 22963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:23.699225 22963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:23.702251 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "83096ecfdb054881aeb1032c4e1c641d"
format_stamp: "Formatted at 2026-09-27 13:49:23 on dist-test-slave-q25r"
server_key: "2706653adca15d29f4345e83c25f0fd9"
server_key_iv: "c14a26a9e5219140131505b44a03959f"
server_key_version: "encryptionkey@0"
I20260927 13:49:23.702992 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "83096ecfdb054881aeb1032c4e1c641d"
format_stamp: "Formatted at 2026-09-27 13:49:23 on dist-test-slave-q25r"
server_key: "2706653adca15d29f4345e83c25f0fd9"
server_key_iv: "c14a26a9e5219140131505b44a03959f"
server_key_version: "encryptionkey@0"
I20260927 13:49:23.708380 22963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:49:23.712122 22978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:23.713615 22963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 13:49:23.713788 22963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "83096ecfdb054881aeb1032c4e1c641d"
format_stamp: "Formatted at 2026-09-27 13:49:23 on dist-test-slave-q25r"
server_key: "2706653adca15d29f4345e83c25f0fd9"
server_key_iv: "c14a26a9e5219140131505b44a03959f"
server_key_version: "encryptionkey@0"
I20260927 13:49:23.713989 22963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:23.727504 22963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:23.731948 22963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:23.732234 22963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:23.734941 22963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:23.735061 22963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:23.735179 22963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:23.735275 22963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:49:23 dist-test-slave-q25r krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516963, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:23.758921 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:23.735431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:35017 (local address 127.20.133.126:43085)
0927 13:49:23.735633 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:23.735639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:23.736998 (+ 1359us) server_negotiation.cc:407] Connection header received
0927 13:49:23.738973 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:23.738979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:23.739078 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:23.739164 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:23.741385 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.742526 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.744984 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.745605 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.745744 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:23.750561 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:23.750569 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:23.750573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:23.750619 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:23.753970 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:23.755688 (+ 1718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:23.755695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:23.755699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:23.755876 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:23.756568 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:23.756573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:23.756575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:23.756979 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:23.757128 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:23.758446 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0927 13:49:23.758659 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:49:23.760789 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:23.735980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43085 (local address 127.20.133.66:35017)
0927 13:49:23.736810 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:23.736867 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:23.738707 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:23.739446 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:23.739459 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:23.740208 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:23.741138 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:23.741167 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.742694 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.742700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:23.744818 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:23.744833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.745873 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.745881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:23.746239 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:23.747115 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:23.747140 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:23.750302 (+ 3162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:23.754187 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:23.754200 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:23.754225 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:23.755408 (+ 1183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:23.756050 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:23.756055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:23.756075 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:23.756433 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:23.757141 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:23.757152 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:23.757891 (+ 739us) client_negotiation.cc:769] Sending connection context
0927 13:49:23.758279 (+ 388us) client_negotiation.cc:240] Negotiation successful
0927 13:49:23.758947 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":377,"threads_started":1}
I20260927 13:49:23.781517 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:23.766749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43085 (local address 127.20.133.66:45951)
0927 13:49:23.767480 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:23.767502 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:23.767705 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:23.768246 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:23.768252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:23.768924 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:23.769372 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:23.769385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.770406 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.770410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:23.771432 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:23.771444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.772166 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.772172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:23.772311 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:23.773135 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:23.773157 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:23.774456 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:23.778598 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:23.778607 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:23.778614 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:23.779133 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:23.779645 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:23.779648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:23.779651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:23.779780 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:23.780615 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:23.780623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:23.780799 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 13:49:23.780915 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 13:49:23.781178 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":348,"threads_started":1}
I20260927 13:49:23.781777 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:23.766809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:45951 (local address 127.20.133.126:43085)
0927 13:49:23.767045 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:23.767052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:23.767651 (+ 599us) server_negotiation.cc:407] Connection header received
0927 13:49:23.767882 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:23.767888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:23.767975 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:23.768084 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:23.769537 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.770253 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.771580 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:23.771973 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:23.772082 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:23.774736 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:23.774745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:23.774751 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:23.774802 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:23.778423 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:23.779295 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:23.779302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:23.779307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:23.779510 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:23.779883 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:23.779888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:23.779893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:23.780462 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:23.780640 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:23.781069 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 13:49:23.781350 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 13:49:23.790627 22963 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:37233
I20260927 13:49:23.790684 23097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:37233 every 8 connection(s)
I20260927 13:49:23.791904 22963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:49:23.793087 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 22963
I20260927 13:49:23.793257 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:49:23.793722 21013 external_mini_cluster.cc:1428] Setting key 0d2c4f10f68b7703de1e74a9e87525f3
I20260927 13:49:23.794189 23098 heartbeater.cc:347] Connected to a master server at 127.20.133.126:43085
I20260927 13:49:23.794401 23098 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:23.795042 23098 heartbeater.cc:511] Master 127.20.133.126:43085 requested a full tablet report, sending...
I20260927 13:49:23.796547 22752 ts_manager.cc:194] Registered new tserver with Master: 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66:37233)
I20260927 13:49:23.797412 22752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:35017
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:49:23.835021 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:43085
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:24.154356 23102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:24.154631 23102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:24.154767 23102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:24.162037 23102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:24.162168 23102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:24.162302 23102 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 13:49:24.162382 23102 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:49:24.162465 23102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:49:24.172386 23102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.133.126:43085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:24.173671 23102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:24.174965 23102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:24.187714 23107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:24.187650 23110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:24.187675 23108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:24.188513 23102 server_base.cc:1034] running on GCE node
I20260927 13:49:24.189093 23102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:24.189988 23102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:24.191171 23102 hybrid_clock.cc:648] HybridClock initialized: now 1790516964191129 us; error 34 us; skew 500 ppm
I20260927 13:49:24.191461 23102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:24 dist-test-slave-q25r krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:24.194432 23102 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:49:24.195824 23102 webserver.cc:460] Webserver started at http://127.20.133.67:40063/ using document root <none> and password file <none>
I20260927 13:49:24.196529 23102 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:24.196648 23102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:24.196933 23102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:24.199792 23102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "eecdfd61dad9417b9ee527949c84246f"
format_stamp: "Formatted at 2026-09-27 13:49:24 on dist-test-slave-q25r"
server_key: "cb9eb0fe523a6fce03482147b485b0ca"
server_key_iv: "1e3cb391e41c54672669bcf9032a5bc1"
server_key_version: "encryptionkey@0"
I20260927 13:49:24.200554 23102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "eecdfd61dad9417b9ee527949c84246f"
format_stamp: "Formatted at 2026-09-27 13:49:24 on dist-test-slave-q25r"
server_key: "cb9eb0fe523a6fce03482147b485b0ca"
server_key_iv: "1e3cb391e41c54672669bcf9032a5bc1"
server_key_version: "encryptionkey@0"
I20260927 13:49:24.206156 23102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:49:24.209834 23117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:24.211278 23102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:49:24.211455 23102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "eecdfd61dad9417b9ee527949c84246f"
format_stamp: "Formatted at 2026-09-27 13:49:24 on dist-test-slave-q25r"
server_key: "cb9eb0fe523a6fce03482147b485b0ca"
server_key_iv: "1e3cb391e41c54672669bcf9032a5bc1"
server_key_version: "encryptionkey@0"
I20260927 13:49:24.211647 23102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:24.248904 23102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:24.253014 23102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:24.253275 23102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:24.255841 23102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:24.255963 23102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:24.256083 23102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:24.256165 23102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:49:24 dist-test-slave-q25r krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:24.281169 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.256596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:52241 (local address 127.20.133.126:43085)
0927 13:49:24.256779 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.256786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.258118 (+ 1332us) server_negotiation.cc:407] Connection header received
0927 13:49:24.260253 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.260260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.260379 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.260490 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:24.262787 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.263828 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.266676 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.267257 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.267385 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.272748 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:24.272755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:24.272760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:24.272801 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:24.276164 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:24.277181 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:24.277188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:24.277192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:24.277367 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:24.277989 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:24.277994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:24.277998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:24.278347 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:24.278496 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.280599 (+ 2103us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.280848 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:49:24.281849 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.257209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43085 (local address 127.20.133.67:52241)
0927 13:49:24.257969 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:24.258010 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:24.259930 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:24.260855 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:24.260866 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:24.261638 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:24.262594 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.262619 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.264002 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.264008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.266501 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.266521 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.267544 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.267555 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.267885 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.268755 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:24.268794 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:24.272434 (+ 3640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:24.276388 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:24.276400 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:24.276425 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:24.277036 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:24.277521 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:24.277526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:24.277529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:24.277816 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:24.278579 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:24.278592 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:24.279180 (+ 588us) client_negotiation.cc:769] Sending connection context
0927 13:49:24.279510 (+ 330us) client_negotiation.cc:240] Negotiation successful
0927 13:49:24.280137 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":324,"threads_started":1}
I20260927 13:49:24.294773 23102 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:42763
I20260927 13:49:24.294813 23236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:42763 every 8 connection(s)
I20260927 13:49:24.295980 23102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:49:24.298269 23237 heartbeater.cc:347] Connected to a master server at 127.20.133.126:43085
I20260927 13:49:24.298491 23237 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:24.299252 23237 heartbeater.cc:511] Master 127.20.133.126:43085 requested a full tablet report, sending...
I20260927 13:49:24.299387 22959 heartbeater.cc:503] Master 127.20.133.126:43085 was elected leader, sending a full tablet report...
I20260927 13:49:24.300797 22751 ts_manager.cc:194] Registered new tserver with Master: eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763)
I20260927 13:49:24.301694 22751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:52241
I20260927 13:49:24.303011 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.289249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43085 (local address 127.20.133.67:43123)
0927 13:49:24.289923 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:24.289945 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:24.290138 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:24.290661 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:24.290666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:24.291345 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:24.291789 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.291806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.293228 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.293231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.294368 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.294380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.295169 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.295176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.295334 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.295887 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:24.295904 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:24.296942 (+ 1038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:24.300401 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:24.300407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:24.300411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:24.300907 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:24.301291 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:24.301296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:24.301300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:24.301427 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:24.301981 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:24.301986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:24.302180 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 13:49:24.302342 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 13:49:24.302635 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":615,"threads_started":1}
I20260927 13:49:24.303279 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 23102
I20260927 13:49:24.303351 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.289666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:43123 (local address 127.20.133.126:43085)
0927 13:49:24.289865 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.289871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.290033 (+ 162us) server_negotiation.cc:407] Connection header received
0927 13:49:24.290291 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.290298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.290388 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.290457 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:24.291971 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.293029 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.294517 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.294984 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.295099 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.297167 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:24.297174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:24.297178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:24.297215 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:24.300250 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:24.301006 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:24.301011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:24.301014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:24.301181 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:24.301533 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:24.301539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:24.301542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:24.301859 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:24.302001 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.302506 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.302768 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 13:49:24.303467 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:49:24.303897 21013 external_mini_cluster.cc:1428] Setting key e1b49ad4781045e429620b6d9eaf9ae0
I20260927 13:49:24.306856 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:49:24 dist-test-slave-q25r krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:49:24 dist-test-slave-q25r krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:24.351547 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.338238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.20.133.126:43085)
0927 13:49:24.338477 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.338484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.338669 (+ 185us) server_negotiation.cc:407] Connection header received
0927 13:49:24.338842 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.338846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.338915 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.338989 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:24.340422 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.341187 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.342425 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.342906 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.343013 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.346675 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:24.346680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:24.346683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:24.346716 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:24.349368 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:24.349971 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:24.349974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:24.349977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:24.350097 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:24.350387 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:24.350389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:24.350391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:24.350653 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:24.350825 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.351053 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.351218 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 13:49:24.355836 22752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:49:24.363348 22752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:49:24.387512 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.375941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:37233 (local address 127.0.0.1:60142)
0927 13:49:24.379495 (+ 3554us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:24.379509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:24.379624 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:24.380511 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:24.380519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:24.380548 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:24.380889 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.380902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.382642 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.382647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.384855 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.384871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.386611 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.386617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.386801 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.386851 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 13:49:24.386958 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 13:49:24.387183 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3440,"thread_start_us":958,"threads_started":1}
I20260927 13:49:24.387492 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.376550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.20.133.66:37233)
0927 13:49:24.378912 (+ 2362us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.378923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.379609 (+ 686us) server_negotiation.cc:407] Connection header received
0927 13:49:24.379778 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.379785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.380031 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.380232 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:24.381116 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.382492 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.385051 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.386443 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.386568 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.386802 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.387038 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.387214 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2178,"thread_start_us":2042,"threads_started":1}
I20260927 13:49:24.388741 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.375870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:41361 (local address 127.0.0.1:51484)
0927 13:49:24.377364 (+ 1494us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:24.377411 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:24.377598 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:24.381608 (+ 4010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:24.381612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:24.381649 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:24.381943 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.381951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.383689 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.383695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.385767 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.385778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.387886 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.387893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.388052 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.388088 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 13:49:24.388179 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 13:49:24.388315 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":1,"thread_start_us":3221,"threads_started":1}
I20260927 13:49:24.389005 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.379667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:42763 (local address 127.0.0.1:47514)
0927 13:49:24.380244 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:24.380261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:24.380377 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:24.382230 (+ 1853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:24.382235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:24.382260 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:24.382576 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.382588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.384106 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.384111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.386347 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.386362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.388136 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.388142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.388321 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.388388 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 13:49:24.388474 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 13:49:24.388598 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":1,"thread_start_us":323,"threads_started":1}
I20260927 13:49:24.389030 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.376047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.20.133.65:41361)
0927 13:49:24.380444 (+ 4397us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.380453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.380853 (+ 400us) server_negotiation.cc:407] Connection header received
0927 13:49:24.380952 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.380959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.381185 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.381365 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:24.382097 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.383558 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.385913 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.387513 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.387643 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.387833 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.388425 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.388588 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4195,"thread_start_us":3212,"threads_started":1}
I20260927 13:49:24.389796 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.380015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.20.133.67:42763)
0927 13:49:24.380957 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.380966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.380996 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:49:24.381083 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.381090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.381304 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.382230 (+ 926us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:24.382771 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.383952 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.386519 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.387911 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.388071 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.388284 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.389311 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.389476 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":600,"threads_started":1}
I20260927 13:49:24.391860 23028 tablet_service.cc:1467] Processing CreateTablet for tablet cad0c9b4de1b413f8a1616d3f9ce82e7 (DEFAULT_TABLE table=test-table [id=496fadab6ce5459f8832c99eeb74e341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:24.392176 22889 tablet_service.cc:1467] Processing CreateTablet for tablet cad0c9b4de1b413f8a1616d3f9ce82e7 (DEFAULT_TABLE table=test-table [id=496fadab6ce5459f8832c99eeb74e341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:24.393136 23028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad0c9b4de1b413f8a1616d3f9ce82e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:24.392879 23167 tablet_service.cc:1467] Processing CreateTablet for tablet cad0c9b4de1b413f8a1616d3f9ce82e7 (DEFAULT_TABLE table=test-table [id=496fadab6ce5459f8832c99eeb74e341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:24.393455 22889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad0c9b4de1b413f8a1616d3f9ce82e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:24.394098 23167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad0c9b4de1b413f8a1616d3f9ce82e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:24.405683 23256 tablet_bootstrap.cc:492] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: Bootstrap starting.
I20260927 13:49:24.408303 23257 tablet_bootstrap.cc:492] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: Bootstrap starting.
I20260927 13:49:24.410046 23258 tablet_bootstrap.cc:492] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: Bootstrap starting.
I20260927 13:49:24.410126 23256 tablet_bootstrap.cc:654] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:24.411546 23256 log.cc:824] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:24.412499 23257 tablet_bootstrap.cc:654] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:24.413158 23258 tablet_bootstrap.cc:654] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:24.413602 23257 log.cc:824] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:24.414203 23258 log.cc:824] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:24.414808 23256 tablet_bootstrap.cc:492] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: No bootstrap required, opened a new log
I20260927 13:49:24.415153 23256 ts_tablet_manager.cc:1397] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 13:49:24.416800 23257 tablet_bootstrap.cc:492] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: No bootstrap required, opened a new log
I20260927 13:49:24.417116 23257 ts_tablet_manager.cc:1397] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260927 13:49:24.417268 23258 tablet_bootstrap.cc:492] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: No bootstrap required, opened a new log
I20260927 13:49:24.417521 23258 ts_tablet_manager.cc:1397] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:49:24.423563 23256 raft_consensus.cc:348] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.423971 23256 raft_consensus.cc:374] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:24.424073 23256 raft_consensus.cc:729] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83096ecfdb054881aeb1032c4e1c641d, State: Initialized, Role: FOLLOWER
I20260927 13:49:24.424710 23256 consensus_queue.cc:260] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.425808 23098 heartbeater.cc:503] Master 127.20.133.126:43085 was elected leader, sending a full tablet report...
I20260927 13:49:24.426308 23256 ts_tablet_manager.cc:1428] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 13:49:24.428180 23257 raft_consensus.cc:348] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.428678 23257 raft_consensus.cc:374] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:24.428854 23257 raft_consensus.cc:729] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d69c7f668e4d8ea28da7355e9d0ed9, State: Initialized, Role: FOLLOWER
I20260927 13:49:24.429376 23258 raft_consensus.cc:348] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.429754 23258 raft_consensus.cc:374] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:24.429610 23257 consensus_queue.cc:260] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.429850 23258 raft_consensus.cc:729] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecdfd61dad9417b9ee527949c84246f, State: Initialized, Role: FOLLOWER
I20260927 13:49:24.430366 23258 consensus_queue.cc:260] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.431911 23257 ts_tablet_manager.cc:1428] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260927 13:49:24.433342 23237 heartbeater.cc:503] Master 127.20.133.126:43085 was elected leader, sending a full tablet report...
I20260927 13:49:24.434167 23258 ts_tablet_manager.cc:1428] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260927 13:49:24.541230 22960 tablet.cc:2369] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:49:24.543991 23099 tablet.cc:2369] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:49:24.547576 23238 tablet.cc:2369] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:49:24.572261 23262 raft_consensus.cc:482] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:49:24.572619 23262 raft_consensus.cc:504] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.574293 23262 leader_election.cc:288] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65:41361), eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763)
I20260927 13:49:24.581256 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.574736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:49445 (local address 127.20.133.67:42763)
0927 13:49:24.574949 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.574956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.575635 (+ 679us) server_negotiation.cc:407] Connection header received
0927 13:49:24.575812 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.575817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.575892 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.575957 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:24.576675 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.577502 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.579463 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.580249 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.580453 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.580533 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.580866 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.580998 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:49:24.581257 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.574841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:46117 (local address 127.20.133.65:41361)
0927 13:49:24.575050 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.575057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.575693 (+ 636us) server_negotiation.cc:407] Connection header received
0927 13:49:24.575862 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.575866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.575938 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.576027 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:24.576741 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.577486 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.579463 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.580238 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.580524 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.580599 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.580889 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.581027 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:49:24.581388 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.574785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:42763 (local address 127.20.133.66:49445)
0927 13:49:24.575515 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:24.575533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:24.575671 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:24.576002 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:24.576007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:24.576030 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:24.576495 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.576510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.577646 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.577649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.579284 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.579297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.580526 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.580533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.580710 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.580735 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 13:49:24.580819 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 13:49:24.580960 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":50,"thread_start_us":339,"threads_started":1}
I20260927 13:49:24.581753 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.574899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:41361 (local address 127.20.133.66:46117)
0927 13:49:24.575575 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:24.575596 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:24.575740 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:24.576057 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:24.576062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:24.576088 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:24.576534 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.576547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.577627 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.577642 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.579290 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.579303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.580559 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.580566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.580734 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.580762 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 13:49:24.580846 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 13:49:24.580982 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":86,"thread_start_us":409,"threads_started":1}
I20260927 13:49:24.582492 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad0c9b4de1b413f8a1616d3f9ce82e7" candidate_uuid: "83096ecfdb054881aeb1032c4e1c641d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecdfd61dad9417b9ee527949c84246f" is_pre_election: true
I20260927 13:49:24.582626 22912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad0c9b4de1b413f8a1616d3f9ce82e7" candidate_uuid: "83096ecfdb054881aeb1032c4e1c641d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" is_pre_election: true
I20260927 13:49:24.583037 23187 raft_consensus.cc:2454] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83096ecfdb054881aeb1032c4e1c641d in term 0.
I20260927 13:49:24.583076 22912 raft_consensus.cc:2454] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83096ecfdb054881aeb1032c4e1c641d in term 0.
I20260927 13:49:24.583730 22982 leader_election.cc:302] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83096ecfdb054881aeb1032c4e1c641d, f6d69c7f668e4d8ea28da7355e9d0ed9; no voters:
I20260927 13:49:24.584167 23262 raft_consensus.cc:2781] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:49:24.584327 23262 raft_consensus.cc:482] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:49:24.584458 23262 raft_consensus.cc:3037] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:24.586879 23262 raft_consensus.cc:504] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.587584 23262 leader_election.cc:288] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [CANDIDATE]: Term 1 election: Requested vote from peers f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65:41361), eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763)
I20260927 13:49:24.588183 22912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad0c9b4de1b413f8a1616d3f9ce82e7" candidate_uuid: "83096ecfdb054881aeb1032c4e1c641d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9"
I20260927 13:49:24.588248 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad0c9b4de1b413f8a1616d3f9ce82e7" candidate_uuid: "83096ecfdb054881aeb1032c4e1c641d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecdfd61dad9417b9ee527949c84246f"
I20260927 13:49:24.588521 23187 raft_consensus.cc:3037] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:24.588510 22912 raft_consensus.cc:3037] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:24.590792 23187 raft_consensus.cc:2454] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83096ecfdb054881aeb1032c4e1c641d in term 1.
I20260927 13:49:24.590907 22912 raft_consensus.cc:2454] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83096ecfdb054881aeb1032c4e1c641d in term 1.
I20260927 13:49:24.591282 22981 leader_election.cc:302] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83096ecfdb054881aeb1032c4e1c641d, eecdfd61dad9417b9ee527949c84246f; no voters:
I20260927 13:49:24.591622 23262 raft_consensus.cc:2781] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:24.592029 23262 raft_consensus.cc:686] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 1 LEADER]: Becoming Leader. State: Replica: 83096ecfdb054881aeb1032c4e1c641d, State: Running, Role: LEADER
I20260927 13:49:24.592576 23262 consensus_queue.cc:237] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.596890 22751 catalog_manager.cc:5495] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66). New cstate: current_term: 1 leader_uuid: "83096ecfdb054881aeb1032c4e1c641d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } health_report { overall_health: UNKNOWN } } }
Sep 27 13:49:24 dist-test-slave-q25r krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516962, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:24.660004 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.646819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.20.133.126:43085)
0927 13:49:24.647001 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.647006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.647482 (+ 476us) server_negotiation.cc:407] Connection header received
0927 13:49:24.647641 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.647646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.647710 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.647789 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:24.648950 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.649715 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.650924 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.651337 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.651443 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.654801 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:24.654806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:24.654809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:24.654837 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:24.657269 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:24.657932 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:24.657938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:24.657942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:24.658066 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:24.658405 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:24.658410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:24.658413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:24.658673 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:24.658821 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.659198 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.659669 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":245,"server-negotiator.queue_time_us":72}
I20260927 13:49:24.660005 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.646833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43085 (local address 127.0.0.1:36754)
0927 13:49:24.647379 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:24.647394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:24.647536 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:24.647833 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:24.647838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:24.648549 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:24.648812 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.648821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.649836 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.649842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.650798 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.650810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.651513 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.651519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.651682 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.652321 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:24.652382 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:24.654618 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:24.657394 (+ 2776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:24.657408 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:24.657418 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:24.657822 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:24.658167 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:24.658172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:24.658175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:24.658308 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:24.658803 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:24.658811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:24.658961 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 13:49:24.659099 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 13:49:24.659494 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":290,"threads_started":1}
I20260927 13:49:24.665302 22751 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:36754:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 13:49:24.666263 22751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:49:24.673269 22889 tablet_service.cc:1467] Processing CreateTablet for tablet 2f459bb710354510955f185cd244f733 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=17daa0f8e9f94bea8157e9f023396dad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:49:24.673418 23028 tablet_service.cc:1467] Processing CreateTablet for tablet 2f459bb710354510955f185cd244f733 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=17daa0f8e9f94bea8157e9f023396dad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:49:24.674284 22889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f459bb710354510955f185cd244f733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:24.674291 23028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f459bb710354510955f185cd244f733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:24.679813 23167 tablet_service.cc:1467] Processing CreateTablet for tablet 2f459bb710354510955f185cd244f733 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=17daa0f8e9f94bea8157e9f023396dad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:49:24.680845 23167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f459bb710354510955f185cd244f733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:24.681506 23256 tablet_bootstrap.cc:492] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d: Bootstrap starting.
I20260927 13:49:24.681821 23257 tablet_bootstrap.cc:492] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: Bootstrap starting.
I20260927 13:49:24.683406 23256 tablet_bootstrap.cc:654] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:24.683430 23257 tablet_bootstrap.cc:654] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:24.688824 23258 tablet_bootstrap.cc:492] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f: Bootstrap starting.
I20260927 13:49:24.690541 23258 tablet_bootstrap.cc:654] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:24.690846 23256 tablet_bootstrap.cc:492] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d: No bootstrap required, opened a new log
I20260927 13:49:24.691017 23256 ts_tablet_manager.cc:1397] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260927 13:49:24.691104 23257 tablet_bootstrap.cc:492] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: No bootstrap required, opened a new log
I20260927 13:49:24.691254 23257 ts_tablet_manager.cc:1397] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260927 13:49:24.691843 23256 raft_consensus.cc:348] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.692102 23256 raft_consensus.cc:374] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:24.691984 23257 raft_consensus.cc:348] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.692198 23256 raft_consensus.cc:729] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83096ecfdb054881aeb1032c4e1c641d, State: Initialized, Role: FOLLOWER
I20260927 13:49:24.692217 23257 raft_consensus.cc:374] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:24.692312 23257 raft_consensus.cc:729] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d69c7f668e4d8ea28da7355e9d0ed9, State: Initialized, Role: FOLLOWER
I20260927 13:49:24.692471 23256 consensus_queue.cc:260] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.692938 23262 tablet_replica.cc:439] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } } }
I20260927 13:49:24.693120 23256 ts_tablet_manager.cc:1428] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 13:49:24.693252 23262 tablet_replica.cc:442] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:24.692627 23257 consensus_queue.cc:260] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.694331 23257 ts_tablet_manager.cc:1428] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 13:49:24.694512 23263 tablet_replica.cc:439] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } } }
I20260927 13:49:24.694775 23263 tablet_replica.cc:442] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:24.700366 23258 tablet_bootstrap.cc:492] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f: No bootstrap required, opened a new log
I20260927 13:49:24.700537 23258 ts_tablet_manager.cc:1397] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260927 13:49:24.701192 23258 raft_consensus.cc:348] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.701398 23258 raft_consensus.cc:374] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:24.701490 23258 raft_consensus.cc:729] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecdfd61dad9417b9ee527949c84246f, State: Initialized, Role: FOLLOWER
I20260927 13:49:24.701714 23258 consensus_queue.cc:260] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.702262 23258 ts_tablet_manager.cc:1428] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 13:49:24.702152 23264 tablet_replica.cc:439] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } } }
I20260927 13:49:24.702510 23264 tablet_replica.cc:442] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:24.901772 23263 raft_consensus.cc:482] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:49:24.902016 23263 raft_consensus.cc:504] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.903600 23263 leader_election.cc:288] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66:37233), eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763)
I20260927 13:49:24.911970 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.904092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:55905 (local address 127.20.133.67:42763)
0927 13:49:24.904269 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.904276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.905413 (+ 1137us) server_negotiation.cc:407] Connection header received
0927 13:49:24.905545 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.905549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.905615 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.905681 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:24.906225 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.907267 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.909679 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.910880 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.911046 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.911123 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.911560 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.911700 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:49:24.912006 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.904217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:42057 (local address 127.20.133.66:37233)
0927 13:49:24.905076 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.905082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.905112 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:49:24.905188 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.905193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.905258 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.905343 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:24.905972 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.907038 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.909680 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.910887 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.911052 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.911122 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.911561 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.911701 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":588,"threads_started":1}
I20260927 13:49:24.912190 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.903939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:37233 (local address 127.20.133.65:42057)
0927 13:49:24.904806 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:24.904828 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:24.904984 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:24.905404 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:24.905409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:24.905431 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:24.905795 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.905811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.907169 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.907172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.909495 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.909508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.911154 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.911160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.911384 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.911431 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 13:49:24.911527 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 13:49:24.911723 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":97,"thread_start_us":1045,"threads_started":1}
I20260927 13:49:24.912532 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.903953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:42763 (local address 127.20.133.65:55905)
0927 13:49:24.905292 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:24.905309 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:24.905424 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:24.905732 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:24.905740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:24.905763 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:24.906068 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.906077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.907367 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.907372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.909492 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.909508 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.911128 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.911139 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.911377 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.911431 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 13:49:24.911527 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 13:49:24.911723 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"mutex_wait_us":131,"thread_start_us":442,"threads_started":1}
I20260927 13:49:24.913141 23048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f459bb710354510955f185cd244f733" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83096ecfdb054881aeb1032c4e1c641d" is_pre_election: true
I20260927 13:49:24.913563 23048 raft_consensus.cc:2454] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 0.
I20260927 13:49:24.914114 22839 leader_election.cc:302] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83096ecfdb054881aeb1032c4e1c641d, f6d69c7f668e4d8ea28da7355e9d0ed9; no voters:
I20260927 13:49:24.914183 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f459bb710354510955f185cd244f733" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecdfd61dad9417b9ee527949c84246f" is_pre_election: true
I20260927 13:49:24.914412 23187 raft_consensus.cc:2454] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 0.
I20260927 13:49:24.914507 23263 raft_consensus.cc:2781] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:49:24.914629 23263 raft_consensus.cc:482] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:49:24.914749 23263 raft_consensus.cc:3037] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:24.917079 23263 raft_consensus.cc:504] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.917681 23263 leader_election.cc:288] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 election: Requested vote from peers 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66:37233), eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763)
I20260927 13:49:24.918154 23048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f459bb710354510955f185cd244f733" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83096ecfdb054881aeb1032c4e1c641d"
I20260927 13:49:24.918166 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f459bb710354510955f185cd244f733" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecdfd61dad9417b9ee527949c84246f"
I20260927 13:49:24.918347 23048 raft_consensus.cc:3037] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:24.918373 23187 raft_consensus.cc:3037] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:24.920575 23187 raft_consensus.cc:2454] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 1.
I20260927 13:49:24.921025 23048 raft_consensus.cc:2454] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 1.
I20260927 13:49:24.921025 22841 leader_election.cc:302] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eecdfd61dad9417b9ee527949c84246f, f6d69c7f668e4d8ea28da7355e9d0ed9; no voters:
I20260927 13:49:24.921478 23263 raft_consensus.cc:2781] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:24.921873 23263 raft_consensus.cc:686] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 LEADER]: Becoming Leader. State: Replica: f6d69c7f668e4d8ea28da7355e9d0ed9, State: Running, Role: LEADER
I20260927 13:49:24.922353 23263 consensus_queue.cc:237] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:24.925454 23281 tablet_replica.cc:439] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: TxnStatusTablet state changed. Reason: New leader f6d69c7f668e4d8ea28da7355e9d0ed9. Latest consensus state: current_term: 1 leader_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } } }
I20260927 13:49:24.926020 23281 tablet_replica.cc:442] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:49:24.926829 23283 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 13:49:24.927107 23283 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 13:49:24.927649 22752 catalog_manager.cc:5495] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } health_report { overall_health: UNKNOWN } } }
I20260927 13:49:24.991298 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.984662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.20.133.65:41361)
0927 13:49:24.985275 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:24.985279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:24.985294 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:49:24.985377 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:24.985380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:24.985467 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:24.985569 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:24.986027 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.986804 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.988827 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.989250 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.989352 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.990026 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:24.990606 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:24.990714 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:24.990909 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 13:49:24.990994 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493}
I20260927 13:49:24.991387 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:24.984638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:41361 (local address 127.0.0.1:51486)
0927 13:49:24.984880 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:24.984897 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:24.985014 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:24.985645 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:24.985651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:24.985667 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:49:24.985885 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.985897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.986932 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.986938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.988675 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:24.988692 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:24.989424 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:24.989431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:24.989684 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:24.989788 (+ 104us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:49:24.990721 (+ 933us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:49:24.990745 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 13:49:24.990846 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 13:49:24.991032 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 13:49:25.000381 23048 raft_consensus.cc:1261] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Refusing update from remote peer f6d69c7f668e4d8ea28da7355e9d0ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:25.000408 23187 raft_consensus.cc:1261] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Refusing update from remote peer f6d69c7f668e4d8ea28da7355e9d0ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:25.001422 23281 consensus_queue.cc:1035] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:25.001735 23263 consensus_queue.cc:1035] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:25.016785 23262 tablet_replica.cc:439] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d: TxnStatusTablet state changed. Reason: New leader f6d69c7f668e4d8ea28da7355e9d0ed9. Latest consensus state: current_term: 1 leader_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } } }
I20260927 13:49:25.017177 23262 tablet_replica.cc:442] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:25.020576 23264 tablet_replica.cc:439] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f: TxnStatusTablet state changed. Reason: New leader f6d69c7f668e4d8ea28da7355e9d0ed9. Latest consensus state: current_term: 1 leader_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } } }
I20260927 13:49:25.020876 23264 tablet_replica.cc:442] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:25.029022 23263 tablet_replica.cc:439] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } } }
I20260927 13:49:25.029575 23263 tablet_replica.cc:442] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:49:25.033118 23262 tablet_replica.cc:439] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } } }
I20260927 13:49:25.033447 23262 tablet_replica.cc:442] T 2f459bb710354510955f185cd244f733 P 83096ecfdb054881aeb1032c4e1c641d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:25.035712 23262 consensus_queue.cc:1035] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:25.038343 23281 tablet_replica.cc:439] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } } }
I20260927 13:49:25.038677 23281 tablet_replica.cc:442] T 2f459bb710354510955f185cd244f733 P f6d69c7f668e4d8ea28da7355e9d0ed9: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:49:25.041153 23288 mvcc.cc:205] Tried to move back new op lower bound from 7333957488630734848 to 7333957488325550080. Current Snapshot: MvccSnapshot[applied={T|T < 7333957488630734848}]
I20260927 13:49:25.043479 23267 consensus_queue.cc:1035] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:25.052500 23264 tablet_replica.cc:439] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } } }
I20260927 13:49:25.052824 23264 tablet_replica.cc:442] T 2f459bb710354510955f185cd244f733 P eecdfd61dad9417b9ee527949c84246f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:49:25.067070 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:25.060173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.20.133.66:37233)
0927 13:49:25.060383 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:25.060390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:25.060908 (+ 518us) server_negotiation.cc:407] Connection header received
0927 13:49:25.061130 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:25.061135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:25.061218 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:25.061346 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:25.061850 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.062701 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.064785 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.065234 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.065343 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:25.065893 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:25.066452 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:25.066567 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:25.066707 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 13:49:25.066796 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 13:49:25.085886 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:25.079192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:54277 (local address 127.20.133.65:41361)
0927 13:49:25.079439 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:25.079446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:25.079828 (+ 382us) server_negotiation.cc:407] Connection header received
0927 13:49:25.080201 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:25.080207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:25.080317 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:25.080607 (+ 290us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:25.081219 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.082087 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.083986 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.084496 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.084619 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:25.085156 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:25.085266 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:25.085378 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:25.085528 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 13:49:25.085622 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:49:25.085968 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:25.079050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:41361 (local address 127.20.133.66:54277)
0927 13:49:25.079722 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:25.079741 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:25.079903 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:25.080633 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:25.080638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:25.080653 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:49:25.081007 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:25.081019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.082250 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.082256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:25.083841 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:25.083857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.084659 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.084665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:25.084856 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:25.084955 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:49:25.085377 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:49:25.085390 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 13:49:25.085474 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 13:49:25.085641 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":349,"threads_started":1}
I20260927 13:49:25.131513 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:25.120990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:37233 (local address 127.20.133.65:51107)
0927 13:49:25.124839 (+ 3849us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:25.124856 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:25.125009 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:25.125340 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:25.125345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:25.125357 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:49:25.125774 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:25.125788 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.127057 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.127073 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:25.129147 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:25.129168 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.130041 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.130050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:25.130257 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:25.130348 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:49:25.130832 (+ 484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:49:25.130852 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 13:49:25.130939 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 13:49:25.131085 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3656,"thread_start_us":3518,"threads_started":1}
I20260927 13:49:25.131513 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:25.121060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:51107 (local address 127.20.133.66:37233)
0927 13:49:25.121227 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:25.121232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:25.124949 (+ 3717us) server_negotiation.cc:407] Connection header received
0927 13:49:25.125152 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:25.125158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:25.125228 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:25.125310 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:25.126005 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.126888 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.129332 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.129835 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.129934 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:25.130565 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:25.130711 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:25.130817 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:25.130962 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 13:49:25.131045 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 13:49:25.176324 22750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:25.176684 22750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:25.180887 22750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468: Sending DeleteTablet for 3 replicas of tablet cad0c9b4de1b413f8a1616d3f9ce82e7
I20260927 13:49:25.182118 22889 tablet_service.cc:1514] Processing DeleteTablet for tablet cad0c9b4de1b413f8a1616d3f9ce82e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:25 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:51484
I20260927 13:49:25.183349 23167 tablet_service.cc:1514] Processing DeleteTablet for tablet cad0c9b4de1b413f8a1616d3f9ce82e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:25 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:47514
I20260927 13:49:25.184558 23028 tablet_service.cc:1514] Processing DeleteTablet for tablet cad0c9b4de1b413f8a1616d3f9ce82e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:25 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:60142
I20260927 13:49:25.188906 23311 tablet_replica.cc:330] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: stopping tablet replica
I20260927 13:49:25.189425 23311 raft_consensus.cc:2229] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:49:25.190740 22751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:49:25.191497 22751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:49:25.193029 23311 raft_consensus.cc:2258] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:25.195218 23311 ts_tablet_manager.cc:1905] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:25.200714 22888 tablet_service.cc:1467] Processing CreateTablet for tablet 0aeb866e14f94ee2ab87cbb739d9c33e (DEFAULT_TABLE table=test-table [id=087c628b6fd84684a912d240a2ceba0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:25.201006 23166 tablet_service.cc:1467] Processing CreateTablet for tablet 0aeb866e14f94ee2ab87cbb739d9c33e (DEFAULT_TABLE table=test-table [id=087c628b6fd84684a912d240a2ceba0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:25.201224 22888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aeb866e14f94ee2ab87cbb739d9c33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:25.201522 23166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aeb866e14f94ee2ab87cbb739d9c33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:25.200119 23028 tablet_service.cc:1467] Processing CreateTablet for tablet 0aeb866e14f94ee2ab87cbb739d9c33e (DEFAULT_TABLE table=test-table [id=087c628b6fd84684a912d240a2ceba0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:25.204908 23028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aeb866e14f94ee2ab87cbb739d9c33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:25.219544 23312 tablet_bootstrap.cc:492] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f: Bootstrap starting.
I20260927 13:49:25.220260 23309 tablet_replica.cc:330] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: stopping tablet replica
I20260927 13:49:25.222036 23311 ts_tablet_manager.cc:1918] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:49:25.222290 23312 tablet_bootstrap.cc:654] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:25.222880 23309 raft_consensus.cc:2229] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:25.223443 23309 raft_consensus.cc:2258] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:25.223475 23311 log.cc:1196] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/cad0c9b4de1b413f8a1616d3f9ce82e7
I20260927 13:49:25.224470 23311 ts_tablet_manager.cc:1939] T cad0c9b4de1b413f8a1616d3f9ce82e7 P 83096ecfdb054881aeb1032c4e1c641d: Deleting consensus metadata
I20260927 13:49:25.225661 23309 ts_tablet_manager.cc:1905] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:25.227708 22736 catalog_manager.cc:4841] TS 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66:37233): tablet cad0c9b4de1b413f8a1616d3f9ce82e7 (table test-table [id=496fadab6ce5459f8832c99eeb74e341]) successfully deleted
I20260927 13:49:25.230173 23310 tablet_replica.cc:330] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: stopping tablet replica
I20260927 13:49:25.230726 23310 raft_consensus.cc:2229] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:25.231482 23314 tablet_bootstrap.cc:492] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9: Bootstrap starting.
I20260927 13:49:25.231943 23310 raft_consensus.cc:2258] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:25.235570 23309 ts_tablet_manager.cc:1918] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:49:25.237205 23313 tablet_bootstrap.cc:492] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d: Bootstrap starting.
I20260927 13:49:25.239367 23309 log.cc:1196] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/cad0c9b4de1b413f8a1616d3f9ce82e7
I20260927 13:49:25.240253 23309 ts_tablet_manager.cc:1939] T cad0c9b4de1b413f8a1616d3f9ce82e7 P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting consensus metadata
I20260927 13:49:25.241078 23313 tablet_bootstrap.cc:654] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:25.241441 23312 tablet_bootstrap.cc:492] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f: No bootstrap required, opened a new log
I20260927 13:49:25.241715 23312 ts_tablet_manager.cc:1397] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20260927 13:49:25.242481 23310 ts_tablet_manager.cc:1905] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:25.242882 23312 raft_consensus.cc:348] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:25.243276 23312 raft_consensus.cc:374] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:25.243470 23312 raft_consensus.cc:729] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecdfd61dad9417b9ee527949c84246f, State: Initialized, Role: FOLLOWER
I20260927 13:49:25.243872 23312 consensus_queue.cc:260] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:25.247231 23312 ts_tablet_manager.cc:1428] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 13:49:25.243784 23314 tablet_bootstrap.cc:654] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:25.250296 22739 catalog_manager.cc:4841] TS f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65:41361): tablet cad0c9b4de1b413f8a1616d3f9ce82e7 (table test-table [id=496fadab6ce5459f8832c99eeb74e341]) successfully deleted
I20260927 13:49:25.255703 23310 ts_tablet_manager.cc:1918] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:49:25.255915 23310 log.cc:1196] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/cad0c9b4de1b413f8a1616d3f9ce82e7
I20260927 13:49:25.256500 23310 ts_tablet_manager.cc:1939] T cad0c9b4de1b413f8a1616d3f9ce82e7 P eecdfd61dad9417b9ee527949c84246f: Deleting consensus metadata
I20260927 13:49:25.258000 23314 tablet_bootstrap.cc:492] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9: No bootstrap required, opened a new log
I20260927 13:49:25.258224 23314 ts_tablet_manager.cc:1397] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.006s
I20260927 13:49:25.258548 22738 catalog_manager.cc:4841] TS eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763): tablet cad0c9b4de1b413f8a1616d3f9ce82e7 (table test-table [id=496fadab6ce5459f8832c99eeb74e341]) successfully deleted
I20260927 13:49:25.259277 23314 raft_consensus.cc:348] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:25.259645 23314 raft_consensus.cc:374] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:25.259802 23314 raft_consensus.cc:729] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d69c7f668e4d8ea28da7355e9d0ed9, State: Initialized, Role: FOLLOWER
I20260927 13:49:25.260145 23314 consensus_queue.cc:260] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:25.260890 23314 ts_tablet_manager.cc:1428] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 13:49:25.260641 23313 tablet_bootstrap.cc:492] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d: No bootstrap required, opened a new log
I20260927 13:49:25.261447 23313 ts_tablet_manager.cc:1397] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20260927 13:49:25.262456 23313 raft_consensus.cc:348] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:25.262715 23313 raft_consensus.cc:374] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:25.262893 23313 raft_consensus.cc:729] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83096ecfdb054881aeb1032c4e1c641d, State: Initialized, Role: FOLLOWER
I20260927 13:49:25.263178 23313 consensus_queue.cc:260] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:25.266371 23313 ts_tablet_manager.cc:1428] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 13:49:25.418241 23307 raft_consensus.cc:482] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:49:25.418501 23307 raft_consensus.cc:504] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:25.419310 23307 leader_election.cc:288] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66:37233), eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763)
I20260927 13:49:25.419992 23048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aeb866e14f94ee2ab87cbb739d9c33e" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83096ecfdb054881aeb1032c4e1c641d" is_pre_election: true
I20260927 13:49:25.420286 23048 raft_consensus.cc:2454] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 0.
I20260927 13:49:25.420836 22839 leader_election.cc:302] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83096ecfdb054881aeb1032c4e1c641d, f6d69c7f668e4d8ea28da7355e9d0ed9; no voters:
I20260927 13:49:25.421177 23307 raft_consensus.cc:2781] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:49:25.421147 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aeb866e14f94ee2ab87cbb739d9c33e" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecdfd61dad9417b9ee527949c84246f" is_pre_election: true
I20260927 13:49:25.421285 23307 raft_consensus.cc:482] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:49:25.421370 23307 raft_consensus.cc:3037] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:25.421376 23187 raft_consensus.cc:2454] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 0.
I20260927 13:49:25.424463 23307 raft_consensus.cc:504] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:25.425283 23307 leader_election.cc:288] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 election: Requested vote from peers 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66:37233), eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763)
I20260927 13:49:25.425606 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aeb866e14f94ee2ab87cbb739d9c33e" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecdfd61dad9417b9ee527949c84246f"
I20260927 13:49:25.425843 23187 raft_consensus.cc:3037] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:25.425887 23048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aeb866e14f94ee2ab87cbb739d9c33e" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83096ecfdb054881aeb1032c4e1c641d"
I20260927 13:49:25.426105 23048 raft_consensus.cc:3037] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:25.428994 23187 raft_consensus.cc:2454] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 1.
I20260927 13:49:25.429569 22841 leader_election.cc:302] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eecdfd61dad9417b9ee527949c84246f, f6d69c7f668e4d8ea28da7355e9d0ed9; no voters:
I20260927 13:49:25.430042 23048 raft_consensus.cc:2454] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 1.
I20260927 13:49:25.430371 23307 raft_consensus.cc:2781] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:25.430856 23307 raft_consensus.cc:686] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 LEADER]: Becoming Leader. State: Replica: f6d69c7f668e4d8ea28da7355e9d0ed9, State: Running, Role: LEADER
I20260927 13:49:25.431160 23307 consensus_queue.cc:237] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:25.434031 22752 catalog_manager.cc:5495] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } health_report { overall_health: UNKNOWN } } }
I20260927 13:49:25.481940 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:25.476614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.20.133.65:41361)
0927 13:49:25.476818 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:25.476824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:25.476927 (+ 103us) server_negotiation.cc:407] Connection header received
0927 13:49:25.477100 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:25.477104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:25.477181 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:25.477253 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:25.477626 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.478326 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.480109 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.480578 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.480685 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:25.481211 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:25.481341 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:25.481421 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:25.481598 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 13:49:25.481667 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:49:25.495673 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:25.490178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:38747 (local address 127.20.133.65:41361)
0927 13:49:25.490434 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:25.490438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:25.490492 (+ 54us) server_negotiation.cc:407] Connection header received
0927 13:49:25.490704 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:25.490709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:25.490778 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:25.490869 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:25.491274 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.492174 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.494038 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.494429 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.494519 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:25.494966 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:25.495075 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:25.495139 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:25.495298 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 13:49:25.495392 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":166}
I20260927 13:49:25.495777 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:25.490193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:41361 (local address 127.20.133.65:38747)
0927 13:49:25.490404 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:25.490423 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:25.490549 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:25.490910 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:25.490914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:25.490926 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:49:25.491137 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:25.491149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.492363 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.492370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:25.493893 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:25.493907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:25.494589 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:25.494595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:25.494758 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:25.494810 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:49:25.495180 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:49:25.495193 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 13:49:25.495290 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 13:49:25.495460 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":21}
I20260927 13:49:25.504077 23186 raft_consensus.cc:1261] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Refusing update from remote peer f6d69c7f668e4d8ea28da7355e9d0ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:25.504163 23048 raft_consensus.cc:1261] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Refusing update from remote peer f6d69c7f668e4d8ea28da7355e9d0ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:25.504704 23307 consensus_queue.cc:1035] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:25.505052 23291 consensus_queue.cc:1035] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:25.508991 23286 mvcc.cc:205] Tried to move back new op lower bound from 7333957490699849728 to 7333957490408321024. Current Snapshot: MvccSnapshot[applied={T|T < 7333957490699849728}]
I20260927 13:49:25.511526 23287 mvcc.cc:205] Tried to move back new op lower bound from 7333957490699849728 to 7333957490408321024. Current Snapshot: MvccSnapshot[applied={T|T < 7333957490699849728}]
W20260927 13:49:25.531677 23286 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 13:49:27.136606 22752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:27.136940 22752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:27.142218 22752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468: Sending DeleteTablet for 3 replicas of tablet 0aeb866e14f94ee2ab87cbb739d9c33e
I20260927 13:49:27.143236 22888 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aeb866e14f94ee2ab87cbb739d9c33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:27 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:51484
I20260927 13:49:27.143386 23167 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aeb866e14f94ee2ab87cbb739d9c33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:27 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:47514
I20260927 13:49:27.143898 23028 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aeb866e14f94ee2ab87cbb739d9c33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:27 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:60142
I20260927 13:49:27.154865 23346 tablet_replica.cc:330] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d: stopping tablet replica
I20260927 13:49:27.155318 23345 tablet_replica.cc:330] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f: stopping tablet replica
I20260927 13:49:27.155354 23346 raft_consensus.cc:2229] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:27.155567 23345 raft_consensus.cc:2229] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:27.155838 23345 raft_consensus.cc:2258] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:27.155893 23346 raft_consensus.cc:2258] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:27.157011 23345 ts_tablet_manager.cc:1905] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:27.157542 23346 ts_tablet_manager.cc:1905] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:27.160866 23344 tablet_replica.cc:330] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9: stopping tablet replica
I20260927 13:49:27.161087 23344 raft_consensus.cc:2229] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:49:27.161747 23344 raft_consensus.cc:2258] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:27.163771 23344 ts_tablet_manager.cc:1905] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:27.164508 23345 ts_tablet_manager.cc:1918] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:49:27.164655 23345 log.cc:1196] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/0aeb866e14f94ee2ab87cbb739d9c33e
I20260927 13:49:27.165030 23345 ts_tablet_manager.cc:1939] T 0aeb866e14f94ee2ab87cbb739d9c33e P eecdfd61dad9417b9ee527949c84246f: Deleting consensus metadata
I20260927 13:49:27.166393 22738 catalog_manager.cc:4841] TS eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763): tablet 0aeb866e14f94ee2ab87cbb739d9c33e (table test-table [id=087c628b6fd84684a912d240a2ceba0b]) successfully deleted
I20260927 13:49:27.168473 23346 ts_tablet_manager.cc:1918] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:49:27.168640 23346 log.cc:1196] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/0aeb866e14f94ee2ab87cbb739d9c33e
I20260927 13:49:27.169035 23346 ts_tablet_manager.cc:1939] T 0aeb866e14f94ee2ab87cbb739d9c33e P 83096ecfdb054881aeb1032c4e1c641d: Deleting consensus metadata
I20260927 13:49:27.170352 22736 catalog_manager.cc:4841] TS 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66:37233): tablet 0aeb866e14f94ee2ab87cbb739d9c33e (table test-table [id=087c628b6fd84684a912d240a2ceba0b]) successfully deleted
I20260927 13:49:27.170562 23344 ts_tablet_manager.cc:1918] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:49:27.170701 23344 log.cc:1196] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/0aeb866e14f94ee2ab87cbb739d9c33e
I20260927 13:49:27.171010 23344 ts_tablet_manager.cc:1939] T 0aeb866e14f94ee2ab87cbb739d9c33e P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting consensus metadata
I20260927 13:49:27.171990 22739 catalog_manager.cc:4841] TS f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65:41361): tablet 0aeb866e14f94ee2ab87cbb739d9c33e (table test-table [id=087c628b6fd84684a912d240a2ceba0b]) successfully deleted
Sep 27 13:49:27 dist-test-slave-q25r krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516967, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:49:27 dist-test-slave-q25r krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516967, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:27.206676 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:27.186994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.20.133.126:43085)
0927 13:49:27.187633 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:27.187640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:27.187669 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:49:27.187897 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:27.187902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:27.187988 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:27.188097 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:27.189648 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:27.190493 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:27.191955 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:27.192485 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:27.192640 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:27.201365 (+ 8725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:27.201372 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:27.201375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:27.201417 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:27.203980 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:27.204774 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:27.204778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:27.204781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:27.204906 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:27.205279 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:27.205282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:27.205285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:27.205576 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:27.205811 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:27.206111 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 13:49:27.206335 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":277,"threads_started":1}
I20260927 13:49:27.211122 22752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:49:27.211733 22752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:49:27.218832 22888 tablet_service.cc:1467] Processing CreateTablet for tablet a81a579ab081456ca3336739352abdc1 (DEFAULT_TABLE table=test-table [id=af88d57f37254942b2bc50ff5bf7d808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:27.218995 23028 tablet_service.cc:1467] Processing CreateTablet for tablet a81a579ab081456ca3336739352abdc1 (DEFAULT_TABLE table=test-table [id=af88d57f37254942b2bc50ff5bf7d808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:27.219101 23167 tablet_service.cc:1467] Processing CreateTablet for tablet a81a579ab081456ca3336739352abdc1 (DEFAULT_TABLE table=test-table [id=af88d57f37254942b2bc50ff5bf7d808]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:27.219367 22888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81a579ab081456ca3336739352abdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:27.219391 23028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81a579ab081456ca3336739352abdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:27.219535 23167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81a579ab081456ca3336739352abdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:27.224849 23355 tablet_bootstrap.cc:492] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f: Bootstrap starting.
I20260927 13:49:27.226282 23357 tablet_bootstrap.cc:492] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d: Bootstrap starting.
I20260927 13:49:27.226634 23355 tablet_bootstrap.cc:654] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:27.226878 23356 tablet_bootstrap.cc:492] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9: Bootstrap starting.
I20260927 13:49:27.228065 23357 tablet_bootstrap.cc:654] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:27.228641 23356 tablet_bootstrap.cc:654] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:27.229182 23355 tablet_bootstrap.cc:492] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f: No bootstrap required, opened a new log
I20260927 13:49:27.229415 23355 ts_tablet_manager.cc:1397] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 13:49:27.230420 23355 raft_consensus.cc:348] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.230770 23355 raft_consensus.cc:374] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:27.230909 23355 raft_consensus.cc:729] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecdfd61dad9417b9ee527949c84246f, State: Initialized, Role: FOLLOWER
I20260927 13:49:27.231230 23355 consensus_queue.cc:260] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.232263 23355 ts_tablet_manager.cc:1428] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 13:49:27.233917 23356 tablet_bootstrap.cc:492] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9: No bootstrap required, opened a new log
I20260927 13:49:27.233918 23357 tablet_bootstrap.cc:492] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d: No bootstrap required, opened a new log
I20260927 13:49:27.234072 23357 ts_tablet_manager.cc:1397] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260927 13:49:27.234072 23356 ts_tablet_manager.cc:1397] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260927 13:49:27.234836 23357 raft_consensus.cc:348] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.234831 23356 raft_consensus.cc:348] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.235074 23356 raft_consensus.cc:374] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:27.235074 23357 raft_consensus.cc:374] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:27.235164 23356 raft_consensus.cc:729] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d69c7f668e4d8ea28da7355e9d0ed9, State: Initialized, Role: FOLLOWER
I20260927 13:49:27.235167 23357 raft_consensus.cc:729] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83096ecfdb054881aeb1032c4e1c641d, State: Initialized, Role: FOLLOWER
I20260927 13:49:27.235428 23357 consensus_queue.cc:260] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.235409 23356 consensus_queue.cc:260] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.235991 23356 ts_tablet_manager.cc:1428] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 13:49:27.236271 23357 ts_tablet_manager.cc:1428] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 13:49:27.255182 23327 raft_consensus.cc:482] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:49:27.255373 23327 raft_consensus.cc:504] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.255982 23327 leader_election.cc:288] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66:37233), eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763)
I20260927 13:49:27.256590 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81a579ab081456ca3336739352abdc1" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecdfd61dad9417b9ee527949c84246f" is_pre_election: true
I20260927 13:49:27.256618 23047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81a579ab081456ca3336739352abdc1" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83096ecfdb054881aeb1032c4e1c641d" is_pre_election: true
I20260927 13:49:27.256834 23187 raft_consensus.cc:2454] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 0.
I20260927 13:49:27.256856 23047 raft_consensus.cc:2454] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 0.
I20260927 13:49:27.257293 22841 leader_election.cc:302] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83096ecfdb054881aeb1032c4e1c641d, eecdfd61dad9417b9ee527949c84246f, f6d69c7f668e4d8ea28da7355e9d0ed9; no voters:
I20260927 13:49:27.257545 23327 raft_consensus.cc:2781] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:49:27.257649 23327 raft_consensus.cc:482] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:49:27.257714 23327 raft_consensus.cc:3037] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:27.259797 23327 raft_consensus.cc:504] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.260392 23327 leader_election.cc:288] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 election: Requested vote from peers 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66:37233), eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763)
I20260927 13:49:27.260811 23047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81a579ab081456ca3336739352abdc1" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83096ecfdb054881aeb1032c4e1c641d"
I20260927 13:49:27.260879 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81a579ab081456ca3336739352abdc1" candidate_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecdfd61dad9417b9ee527949c84246f"
I20260927 13:49:27.261027 23047 raft_consensus.cc:3037] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:27.261051 23187 raft_consensus.cc:3037] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:27.263892 23047 raft_consensus.cc:2454] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 1.
I20260927 13:49:27.263892 23187 raft_consensus.cc:2454] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d69c7f668e4d8ea28da7355e9d0ed9 in term 1.
I20260927 13:49:27.264300 22841 leader_election.cc:302] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83096ecfdb054881aeb1032c4e1c641d, eecdfd61dad9417b9ee527949c84246f, f6d69c7f668e4d8ea28da7355e9d0ed9; no voters:
I20260927 13:49:27.264633 23327 raft_consensus.cc:2781] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:27.264784 23327 raft_consensus.cc:686] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 LEADER]: Becoming Leader. State: Replica: f6d69c7f668e4d8ea28da7355e9d0ed9, State: Running, Role: LEADER
I20260927 13:49:27.265064 23327 consensus_queue.cc:237] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.267772 22751 catalog_manager.cc:5495] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } health_report { overall_health: UNKNOWN } } }
I20260927 13:49:27.299853 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:27.292970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.20.133.65:41361)
0927 13:49:27.293590 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:27.293595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:27.293621 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:49:27.293687 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:27.293692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:27.293779 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:27.293888 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:27.294336 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:27.295505 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:27.297445 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:27.298196 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:27.298333 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:27.298894 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:27.299078 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:27.299176 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:27.299333 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 13:49:27.299421 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":353,"threads_started":1}
I20260927 13:49:27.310315 23047 raft_consensus.cc:1261] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Refusing update from remote peer f6d69c7f668e4d8ea28da7355e9d0ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:27.310520 23187 raft_consensus.cc:1261] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Refusing update from remote peer f6d69c7f668e4d8ea28da7355e9d0ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:27.311121 23263 consensus_queue.cc:1035] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:27.311690 23263 consensus_queue.cc:1035] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:27.363788 22751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:27.364033 22751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:27.366971 22751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468: Sending DeleteTablet for 3 replicas of tablet a81a579ab081456ca3336739352abdc1
I20260927 13:49:27.367779 22889 tablet_service.cc:1514] Processing DeleteTablet for tablet a81a579ab081456ca3336739352abdc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:27 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:51484
I20260927 13:49:27.367859 23028 tablet_service.cc:1514] Processing DeleteTablet for tablet a81a579ab081456ca3336739352abdc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:27 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:60142
I20260927 13:49:27.367934 23167 tablet_service.cc:1514] Processing DeleteTablet for tablet a81a579ab081456ca3336739352abdc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:27 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:47514
I20260927 13:49:27.368074 23344 tablet_replica.cc:330] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9: stopping tablet replica
I20260927 13:49:27.368191 23345 tablet_replica.cc:330] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f: stopping tablet replica
I20260927 13:49:27.368295 23346 tablet_replica.cc:330] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d: stopping tablet replica
I20260927 13:49:27.368290 23344 raft_consensus.cc:2229] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:49:27.368415 23345 raft_consensus.cc:2229] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:27.368527 23346 raft_consensus.cc:2229] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:27.368603 23345 raft_consensus.cc:2258] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:27.368650 23344 raft_consensus.cc:2258] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:27.368721 23346 raft_consensus.cc:2258] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:27.369299 22751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 13:49:27.369673 23345 ts_tablet_manager.cc:1905] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:27.369696 23344 ts_tablet_manager.cc:1905] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 13:49:27.369886 22751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:49:27.371655 23346 ts_tablet_manager.cc:1905] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:27.376039 23345 ts_tablet_manager.cc:1918] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:49:27.376039 23344 ts_tablet_manager.cc:1918] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:49:27.376200 23345 log.cc:1196] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/a81a579ab081456ca3336739352abdc1
I20260927 13:49:27.376653 23345 ts_tablet_manager.cc:1939] T a81a579ab081456ca3336739352abdc1 P eecdfd61dad9417b9ee527949c84246f: Deleting consensus metadata
I20260927 13:49:27.377028 22889 tablet_service.cc:1467] Processing CreateTablet for tablet 8f316e6e82d94fc087e6914e2c542d39 (DEFAULT_TABLE table=test-table [id=24aa039a687a474091e66bb8b3af9844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:27.377437 22889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f316e6e82d94fc087e6914e2c542d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:27.377933 23346 ts_tablet_manager.cc:1918] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:49:27.377875 23167 tablet_service.cc:1467] Processing CreateTablet for tablet 8f316e6e82d94fc087e6914e2c542d39 (DEFAULT_TABLE table=test-table [id=24aa039a687a474091e66bb8b3af9844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:27.378067 23346 log.cc:1196] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/a81a579ab081456ca3336739352abdc1
I20260927 13:49:27.378257 23167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f316e6e82d94fc087e6914e2c542d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:27.378440 23346 ts_tablet_manager.cc:1939] T a81a579ab081456ca3336739352abdc1 P 83096ecfdb054881aeb1032c4e1c641d: Deleting consensus metadata
I20260927 13:49:27.379356 22738 catalog_manager.cc:4841] TS eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763): tablet a81a579ab081456ca3336739352abdc1 (table test-table [id=af88d57f37254942b2bc50ff5bf7d808]) successfully deleted
I20260927 13:49:27.380232 23028 tablet_service.cc:1467] Processing CreateTablet for tablet 8f316e6e82d94fc087e6914e2c542d39 (DEFAULT_TABLE table=test-table [id=24aa039a687a474091e66bb8b3af9844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:27.380386 22736 catalog_manager.cc:4841] TS 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66:37233): tablet a81a579ab081456ca3336739352abdc1 (table test-table [id=af88d57f37254942b2bc50ff5bf7d808]) successfully deleted
I20260927 13:49:27.380631 23028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f316e6e82d94fc087e6914e2c542d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:27.382093 23344 log.cc:1196] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/a81a579ab081456ca3336739352abdc1
I20260927 13:49:27.382226 23356 tablet_bootstrap.cc:492] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9: Bootstrap starting.
I20260927 13:49:27.382688 23344 ts_tablet_manager.cc:1939] T a81a579ab081456ca3336739352abdc1 P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting consensus metadata
I20260927 13:49:27.383916 23356 tablet_bootstrap.cc:654] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:27.384040 23355 tablet_bootstrap.cc:492] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f: Bootstrap starting.
I20260927 13:49:27.385599 23355 tablet_bootstrap.cc:654] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:27.385623 22739 catalog_manager.cc:4841] TS f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65:41361): tablet a81a579ab081456ca3336739352abdc1 (table test-table [id=af88d57f37254942b2bc50ff5bf7d808]) successfully deleted
I20260927 13:49:27.386744 23357 tablet_bootstrap.cc:492] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d: Bootstrap starting.
I20260927 13:49:27.388160 23356 tablet_bootstrap.cc:492] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9: No bootstrap required, opened a new log
I20260927 13:49:27.388176 23355 tablet_bootstrap.cc:492] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f: No bootstrap required, opened a new log
I20260927 13:49:27.388262 23357 tablet_bootstrap.cc:654] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:27.388327 23355 ts_tablet_manager.cc:1397] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 13:49:27.388491 23356 ts_tablet_manager.cc:1397] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 13:49:27.389060 23355 raft_consensus.cc:348] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.389160 23356 raft_consensus.cc:348] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.389335 23355 raft_consensus.cc:374] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:27.389369 23356 raft_consensus.cc:374] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:27.389415 23355 raft_consensus.cc:729] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecdfd61dad9417b9ee527949c84246f, State: Initialized, Role: FOLLOWER
I20260927 13:49:27.389444 23356 raft_consensus.cc:729] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d69c7f668e4d8ea28da7355e9d0ed9, State: Initialized, Role: FOLLOWER
I20260927 13:49:27.389652 23355 consensus_queue.cc:260] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.389686 23356 consensus_queue.cc:260] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.390041 23357 tablet_bootstrap.cc:492] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d: No bootstrap required, opened a new log
I20260927 13:49:27.390172 23356 ts_tablet_manager.cc:1428] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 13:49:27.390197 23357 ts_tablet_manager.cc:1397] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 13:49:27.390722 23355 ts_tablet_manager.cc:1428] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:49:27.390884 23357 raft_consensus.cc:348] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.391130 23357 raft_consensus.cc:374] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:27.391219 23357 raft_consensus.cc:729] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83096ecfdb054881aeb1032c4e1c641d, State: Initialized, Role: FOLLOWER
I20260927 13:49:27.391448 23357 consensus_queue.cc:260] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.392280 23357 ts_tablet_manager.cc:1428] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:49:27.537335 23362 raft_consensus.cc:482] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:49:27.537568 23362 raft_consensus.cc:504] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.538262 23362 leader_election.cc:288] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65:41361), eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763)
I20260927 13:49:27.538841 22912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f316e6e82d94fc087e6914e2c542d39" candidate_uuid: "83096ecfdb054881aeb1032c4e1c641d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" is_pre_election: true
I20260927 13:49:27.538861 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f316e6e82d94fc087e6914e2c542d39" candidate_uuid: "83096ecfdb054881aeb1032c4e1c641d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecdfd61dad9417b9ee527949c84246f" is_pre_election: true
I20260927 13:49:27.539077 23187 raft_consensus.cc:2454] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83096ecfdb054881aeb1032c4e1c641d in term 0.
I20260927 13:49:27.539078 22912 raft_consensus.cc:2454] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83096ecfdb054881aeb1032c4e1c641d in term 0.
I20260927 13:49:27.539500 22981 leader_election.cc:302] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83096ecfdb054881aeb1032c4e1c641d, eecdfd61dad9417b9ee527949c84246f; no voters:
I20260927 13:49:27.539767 23362 raft_consensus.cc:2781] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:49:27.539876 23362 raft_consensus.cc:482] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:49:27.539975 23362 raft_consensus.cc:3037] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:27.542183 23362 raft_consensus.cc:504] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.542697 23362 leader_election.cc:288] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [CANDIDATE]: Term 1 election: Requested vote from peers f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65:41361), eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763)
I20260927 13:49:27.543217 22912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f316e6e82d94fc087e6914e2c542d39" candidate_uuid: "83096ecfdb054881aeb1032c4e1c641d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9"
I20260927 13:49:27.543247 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f316e6e82d94fc087e6914e2c542d39" candidate_uuid: "83096ecfdb054881aeb1032c4e1c641d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecdfd61dad9417b9ee527949c84246f"
I20260927 13:49:27.543481 22912 raft_consensus.cc:3037] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:27.543488 23187 raft_consensus.cc:3037] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:27.546407 23187 raft_consensus.cc:2454] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83096ecfdb054881aeb1032c4e1c641d in term 1.
I20260927 13:49:27.546393 22912 raft_consensus.cc:2454] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83096ecfdb054881aeb1032c4e1c641d in term 1.
I20260927 13:49:27.546865 22981 leader_election.cc:302] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83096ecfdb054881aeb1032c4e1c641d, eecdfd61dad9417b9ee527949c84246f, f6d69c7f668e4d8ea28da7355e9d0ed9; no voters:
I20260927 13:49:27.547168 23362 raft_consensus.cc:2781] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:27.547623 23362 raft_consensus.cc:686] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 1 LEADER]: Becoming Leader. State: Replica: 83096ecfdb054881aeb1032c4e1c641d, State: Running, Role: LEADER
I20260927 13:49:27.547921 23362 consensus_queue.cc:237] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } }
I20260927 13:49:27.550504 22751 catalog_manager.cc:5495] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66). New cstate: current_term: 1 leader_uuid: "83096ecfdb054881aeb1032c4e1c641d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83096ecfdb054881aeb1032c4e1c641d" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 } health_report { overall_health: UNKNOWN } } }
I20260927 13:49:27.595880 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:27.590479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.20.133.66:37233)
0927 13:49:27.591015 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:27.591019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:27.591038 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:49:27.591089 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:27.591093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:27.591153 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:27.591237 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:27.591759 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:27.592629 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:27.594033 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:27.594635 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:27.594737 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:27.595236 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:27.595349 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:27.595416 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:27.595527 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 13:49:27.595614 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
I20260927 13:49:27.606487 23187 raft_consensus.cc:1261] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Refusing update from remote peer 83096ecfdb054881aeb1032c4e1c641d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:27.606421 22912 raft_consensus.cc:1261] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Refusing update from remote peer 83096ecfdb054881aeb1032c4e1c641d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:27.607504 23372 consensus_queue.cc:1035] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eecdfd61dad9417b9ee527949c84246f" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:27.607950 23362 consensus_queue.cc:1035] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d69c7f668e4d8ea28da7355e9d0ed9" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 13:49:27.627024 23329 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 13:49:29.208956 22751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:29.209228 22751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:29.213219 22751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e59d5973ba96416b8e6c75f7bdc48468: Sending DeleteTablet for 3 replicas of tablet 8f316e6e82d94fc087e6914e2c542d39
I20260927 13:49:29.213989 22889 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f316e6e82d94fc087e6914e2c542d39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:29 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:51484
I20260927 13:49:29.214063 23028 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f316e6e82d94fc087e6914e2c542d39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:29 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:60142
I20260927 13:49:29.214152 23167 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f316e6e82d94fc087e6914e2c542d39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:29 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:47514
I20260927 13:49:29.215016 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22823
I20260927 13:49:29.226629 23401 tablet_replica.cc:330] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9: stopping tablet replica
I20260927 13:49:29.226660 23403 tablet_replica.cc:330] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f: stopping tablet replica
I20260927 13:49:29.226859 23401 raft_consensus.cc:2229] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:29.226898 23403 raft_consensus.cc:2229] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:29.227164 23401 raft_consensus.cc:2258] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:29.227324 23403 raft_consensus.cc:2258] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:29.228456 23401 ts_tablet_manager.cc:1905] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:29.228835 23403 ts_tablet_manager.cc:1905] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:29.235644 23403 ts_tablet_manager.cc:1918] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:49:29.235888 23403 log.cc:1196] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/8f316e6e82d94fc087e6914e2c542d39
I20260927 13:49:29.236032 23401 ts_tablet_manager.cc:1918] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:49:29.236241 23401 log.cc:1196] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/8f316e6e82d94fc087e6914e2c542d39
I20260927 13:49:29.236480 23403 ts_tablet_manager.cc:1939] T 8f316e6e82d94fc087e6914e2c542d39 P eecdfd61dad9417b9ee527949c84246f: Deleting consensus metadata
I20260927 13:49:29.236912 23401 ts_tablet_manager.cc:1939] T 8f316e6e82d94fc087e6914e2c542d39 P f6d69c7f668e4d8ea28da7355e9d0ed9: Deleting consensus metadata
I20260927 13:49:29.237898 22738 catalog_manager.cc:4841] TS eecdfd61dad9417b9ee527949c84246f (127.20.133.67:42763): tablet 8f316e6e82d94fc087e6914e2c542d39 (table test-table [id=24aa039a687a474091e66bb8b3af9844]) successfully deleted
I20260927 13:49:29.238229 22739 catalog_manager.cc:4841] TS f6d69c7f668e4d8ea28da7355e9d0ed9 (127.20.133.65:41361): tablet 8f316e6e82d94fc087e6914e2c542d39 (table test-table [id=24aa039a687a474091e66bb8b3af9844]) successfully deleted
I20260927 13:49:29.238536 23402 tablet_replica.cc:330] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d: stopping tablet replica
I20260927 13:49:29.238730 23402 raft_consensus.cc:2229] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:49:29.239058 23402 raft_consensus.cc:2258] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 13:49:29 dist-test-slave-q25r krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516967, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:29.239898 23402 ts_tablet_manager.cc:1905] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:29.244446 23402 ts_tablet_manager.cc:1918] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:49:29.244592 23402 log.cc:1196] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/8f316e6e82d94fc087e6914e2c542d39
I20260927 13:49:29.244973 23402 ts_tablet_manager.cc:1939] T 8f316e6e82d94fc087e6914e2c542d39 P 83096ecfdb054881aeb1032c4e1c641d: Deleting consensus metadata
I20260927 13:49:29.245937 22736 catalog_manager.cc:4841] TS 83096ecfdb054881aeb1032c4e1c641d (127.20.133.66:37233): tablet 8f316e6e82d94fc087e6914e2c542d39 (table test-table [id=24aa039a687a474091e66bb8b3af9844]) successfully deleted
I20260927 13:49:29.246388 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:29.217045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.20.133.65:41361)
0927 13:49:29.222429 (+ 5384us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:29.222436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:29.229189 (+ 6753us) server_negotiation.cc:407] Connection header received
0927 13:49:29.229470 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:29.229475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:29.229632 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:29.229777 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:29.231548 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:29.232675 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:29.234499 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:29.235162 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:29.235277 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:29.240639 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:29.240655 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:29.240672 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:29.240706 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:29.243551 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:29.244231 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:29.244238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:29.244241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:29.244416 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:29.244858 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:29.244864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:29.244867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:29.245246 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:29.245526 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:29.245821 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 13:49:29.246048 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5237,"thread_start_us":5050,"threads_started":1}
W20260927 13:49:29.246975 22954 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51522
W20260927 13:49:29.247562 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22823: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:49:29.247780 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 22823
I20260927 13:49:29.269291 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22963
Sep 27 13:49:29 dist-test-slave-q25r krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516967, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:49:29.285782 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:29.270026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.20.133.66:37233)
0927 13:49:29.270502 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:29.270506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:29.270524 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:49:29.270579 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:29.270583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:29.270650 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:29.270758 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:29.272058 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:29.273184 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:29.274529 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:29.274993 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:29.275114 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:29.279592 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:29.279613 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:29.279636 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:29.279687 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:29.282781 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:29.283518 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:29.283525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:29.283528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:29.283708 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:29.284132 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:29.284138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:29.284141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:29.284669 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:29.284953 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:29.285286 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 13:49:29.285484 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":255,"threads_started":1}
W20260927 13:49:29.286440 23093 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60158
W20260927 13:49:29.287072 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22963: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:49:29.287214 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 22963
I20260927 13:49:29.304528 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 23102
Sep 27 13:49:29 dist-test-slave-q25r krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516967, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:49:29.321452 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:29.305293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.20.133.67:42763)
0927 13:49:29.305976 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:29.305980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:29.305995 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:49:29.306044 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:29.306047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:29.306111 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:29.306195 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:29.307923 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:29.308801 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:29.310532 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:29.310995 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:29.311086 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:29.315523 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:29.315539 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:29.315556 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:29.315593 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:29.318377 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:29.319197 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:29.319206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:29.319208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:29.319376 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:29.319875 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:29.319881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:29.319884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:29.320292 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:29.320570 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:29.320873 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 13:49:29.321101 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":427,"threads_started":1}
W20260927 13:49:29.322086 23232 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47522
W20260927 13:49:29.322686 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 23102: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:49:29.322839 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 23102
I20260927 13:49:29.336465 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 22720
I20260927 13:49:29.337700 22792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:29.455230 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 22720
2026-09-27T13:49:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8451 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:49:29 dist-test-slave-q25r krb5kdc[23411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:49:29 dist-test-slave-q25r krb5kdc[23411](info): set up 2 sockets
Sep 27 13:49:29 dist-test-slave-q25r krb5kdc[23411](info): commencing operation
krb5kdc: starting...
W20260927 13:49:30.469071 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:49:30 dist-test-slave-q25r krb5kdc[23411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:49:30.687685 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:32913
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:32913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:31.068600 23424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:31.069048 23424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:31.069198 23424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:31.077123 23424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:31.077279 23424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:31.077363 23424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:31.077443 23424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:31.077509 23424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:31.088716 23424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:32913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:32913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:31.090559 23424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:31.092245 23424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:31.108947 23429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:31.109037 23432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:31.108975 23430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:31.110122 23424 server_base.cc:1034] running on GCE node
I20260927 13:49:31.111188 23424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:31.113114 23424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:31.114346 23424 hybrid_clock.cc:648] HybridClock initialized: now 1790516971114270 us; error 64 us; skew 500 ppm
I20260927 13:49:31.114794 23424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:31 dist-test-slave-q25r krb5kdc[23411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516971, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:31.118602 23424 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:49:31.120579 23424 webserver.cc:460] Webserver started at http://127.20.133.126:36809/ using document root <none> and password file <none>
I20260927 13:49:31.121469 23424 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:31.121680 23424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:31.122083 23424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:31.125870 23424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "29f760cfc5e24b77a0c0ec2cc6bf52e0"
format_stamp: "Formatted at 2026-09-27 13:49:31 on dist-test-slave-q25r"
server_key: "2a52074dd5eb4168cfe423e5c378a499"
server_key_iv: "ed7f485c343aad0c022d8fd199b0fda5"
server_key_version: "encryptionkey@0"
I20260927 13:49:31.126713 23424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "29f760cfc5e24b77a0c0ec2cc6bf52e0"
format_stamp: "Formatted at 2026-09-27 13:49:31 on dist-test-slave-q25r"
server_key: "2a52074dd5eb4168cfe423e5c378a499"
server_key_iv: "ed7f485c343aad0c022d8fd199b0fda5"
server_key_version: "encryptionkey@0"
I20260927 13:49:31.133139 23424 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 13:49:31.137454 23439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:31.139154 23424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:49:31.139348 23424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "29f760cfc5e24b77a0c0ec2cc6bf52e0"
format_stamp: "Formatted at 2026-09-27 13:49:31 on dist-test-slave-q25r"
server_key: "2a52074dd5eb4168cfe423e5c378a499"
server_key_iv: "ed7f485c343aad0c022d8fd199b0fda5"
server_key_version: "encryptionkey@0"
I20260927 13:49:31.139571 23424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:31.175220 23424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:31.180166 23424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:31.180506 23424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:31.201833 23424 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:32913
I20260927 13:49:31.201870 23490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:32913 every 8 connection(s)
I20260927 13:49:31.203544 23424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:31.207901 23491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:31.212244 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 23424
I20260927 13:49:31.212540 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:31.213325 21013 external_mini_cluster.cc:1428] Setting key 00782d67ffc16b42e5ce09cfe9528eb3
I20260927 13:49:31.220697 23491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0: Bootstrap starting.
I20260927 13:49:31.224927 23491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:31.226202 23491 log.cc:824] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:31.230201 23491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0: No bootstrap required, opened a new log
Sep 27 13:49:31 dist-test-slave-q25r krb5kdc[23411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:31.238511 23491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f760cfc5e24b77a0c0ec2cc6bf52e0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32913 } }
I20260927 13:49:31.239060 23491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:31.239183 23491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f760cfc5e24b77a0c0ec2cc6bf52e0, State: Initialized, Role: FOLLOWER
I20260927 13:49:31.239923 23491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f760cfc5e24b77a0c0ec2cc6bf52e0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32913 } }
I20260927 13:49:31.240206 23491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:31.240383 23491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:31.240571 23491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:31.240660 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:31.215925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38014 (local address 127.20.133.126:32913)
0927 13:49:31.216936 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:31.216951 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:31.217007 (+ 56us) server_negotiation.cc:407] Connection header received
0927 13:49:31.219206 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:31.219236 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:31.219744 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:31.220165 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:31.222081 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:31.223699 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:31.225630 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:31.226602 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:31.226804 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:31.231700 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:31.231710 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:31.231733 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:31.231789 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:31.235418 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:31.236299 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:31.236307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:31.236316 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:31.236584 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:31.237004 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:31.237008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:31.237010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:31.237753 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:31.238057 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:31.238591 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 13:49:31.239120 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":467,"threads_started":1}
I20260927 13:49:31.243237 23491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f760cfc5e24b77a0c0ec2cc6bf52e0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32913 } }
I20260927 13:49:31.243947 23491 leader_election.cc:302] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f760cfc5e24b77a0c0ec2cc6bf52e0; no voters:
I20260927 13:49:31.244660 23491 leader_election.cc:288] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:31.244830 23496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:31.245664 23496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [term 1 LEADER]: Becoming Leader. State: Replica: 29f760cfc5e24b77a0c0ec2cc6bf52e0, State: Running, Role: LEADER
I20260927 13:49:31.246316 23496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f760cfc5e24b77a0c0ec2cc6bf52e0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32913 } }
I20260927 13:49:31.247022 23491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:31.249274 23497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29f760cfc5e24b77a0c0ec2cc6bf52e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f760cfc5e24b77a0c0ec2cc6bf52e0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32913 } } }
I20260927 13:49:31.249752 23497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:31.249630 23498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f760cfc5e24b77a0c0ec2cc6bf52e0. Latest consensus state: current_term: 1 leader_uuid: "29f760cfc5e24b77a0c0ec2cc6bf52e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f760cfc5e24b77a0c0ec2cc6bf52e0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32913 } } }
I20260927 13:49:31.249930 23498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:31.250670 23501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:31.257210 23501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:31.269132 23501 catalog_manager.cc:1347] Generated new cluster ID: 5e4dab022af14f42963de4262ec7d196
I20260927 13:49:31.269279 23501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:31.286175 23501 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:31.288157 23501 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:31.295763 23501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29f760cfc5e24b77a0c0ec2cc6bf52e0: Generated new TSK 0
I20260927 13:49:31.296949 23501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:49:31.345218 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32913
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:31.697927 23517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:31.698244 23517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:31.698349 23517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:31.706248 23517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:31.706384 23517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:31.706499 23517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:49:31.718171 23517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:31.719472 23517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:31.720818 23517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:31.733906 23522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:31.733969 23525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:31.733922 23523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:31.734802 23517 server_base.cc:1034] running on GCE node
I20260927 13:49:31.735442 23517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:31.736495 23517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:31.737757 23517 hybrid_clock.cc:648] HybridClock initialized: now 1790516971737697 us; error 49 us; skew 500 ppm
I20260927 13:49:31.738099 23517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:31 dist-test-slave-q25r krb5kdc[23411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516971, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:31.741696 23517 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:49:31.743325 23517 webserver.cc:460] Webserver started at http://127.20.133.65:45007/ using document root <none> and password file <none>
I20260927 13:49:31.744091 23517 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:31.744200 23517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:31.744534 23517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:31.747687 23517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "48f7cd947dd4475eb67f26cb801dcf39"
format_stamp: "Formatted at 2026-09-27 13:49:31 on dist-test-slave-q25r"
server_key: "5a7f2bb0268f26cc088ad8708fe85d08"
server_key_iv: "04f529158d6ea47f98eb8bce12a89d9a"
server_key_version: "encryptionkey@0"
I20260927 13:49:31.748524 23517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "48f7cd947dd4475eb67f26cb801dcf39"
format_stamp: "Formatted at 2026-09-27 13:49:31 on dist-test-slave-q25r"
server_key: "5a7f2bb0268f26cc088ad8708fe85d08"
server_key_iv: "04f529158d6ea47f98eb8bce12a89d9a"
server_key_version: "encryptionkey@0"
I20260927 13:49:31.754809 23517 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 13:49:31.758373 23532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:31.759742 23517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:49:31.759881 23517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "48f7cd947dd4475eb67f26cb801dcf39"
format_stamp: "Formatted at 2026-09-27 13:49:31 on dist-test-slave-q25r"
server_key: "5a7f2bb0268f26cc088ad8708fe85d08"
server_key_iv: "04f529158d6ea47f98eb8bce12a89d9a"
server_key_version: "encryptionkey@0"
I20260927 13:49:31.760061 23517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:31.773020 23517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:31.777930 23517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:31.778239 23517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:31.779241 23517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:31.780828 23517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:31.780918 23517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:31.781016 23517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:31.781075 23517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:31.815492 23517 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:35695
I20260927 13:49:31.815517 23644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:35695 every 8 connection(s)
I20260927 13:49:31.817020 23517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:49:31.824828 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 23517
I20260927 13:49:31.825037 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:49:31.825682 21013 external_mini_cluster.cc:1428] Setting key 7055019a0ca50ce622a0f25aa5c27722
Sep 27 13:49:31 dist-test-slave-q25r krb5kdc[23411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516971, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:31.840075 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:31.820090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:39551 (local address 127.20.133.126:32913)
0927 13:49:31.820613 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:31.820620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:31.821693 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 13:49:31.823526 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:31.823531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:31.823605 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:31.823710 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:31.825662 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:31.826629 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:31.828519 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:31.829109 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:31.829265 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:31.833367 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:31.833375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:31.833379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:31.833424 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:31.836431 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:31.837383 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:31.837390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:31.837394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:31.837551 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:31.837999 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:31.838003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:31.838007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:31.838354 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:31.838504 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:31.839397 (+ 893us) server_negotiation.cc:299] Negotiation successful
0927 13:49:31.839674 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":280,"threads_started":1}
I20260927 13:49:31.841085 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:31.820714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:32913 (local address 127.20.133.65:39551)
0927 13:49:31.821503 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:31.821551 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:31.823226 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:31.823919 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:31.823928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:31.824558 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:31.825437 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:31.825461 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:31.826839 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:31.826844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:31.828327 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:31.828379 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:31.829418 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:31.829424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:31.829683 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:31.830476 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:31.830505 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:31.833020 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:31.836660 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:31.836669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:31.836686 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:31.837230 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:31.837652 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:31.837655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:31.837658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:31.837877 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:31.838530 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:31.838539 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:31.838927 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 13:49:31.839286 (+ 359us) client_negotiation.cc:240] Negotiation successful
0927 13:49:31.839858 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":306,"threads_started":1}
I20260927 13:49:31.842712 23645 heartbeater.cc:347] Connected to a master server at 127.20.133.126:32913
I20260927 13:49:31.843051 23645 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
I20260927 13:49:31.843859 23645 heartbeater.cc:511] Master 127.20.133.126:32913 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
I20260927 13:49:31.846222 23456 ts_manager.cc:194] Registered new tserver with Master: 48f7cd947dd4475eb67f26cb801dcf39 (127.20.133.65:35695)
I20260927 13:49:31.848320 23456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:39551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:49:31.871615 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32913
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:32.212646 23651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:32.212951 23651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:32.213088 23651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:32.220575 23651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:32.220718 23651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:32.220870 23651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:49:32.230923 23651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:32.232266 23651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:32.233618 23651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:32.251952 23657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:32.254130 23656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:32.254532 23659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:32.254566 23651 server_base.cc:1034] running on GCE node
I20260927 13:49:32.255405 23651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:32.256524 23651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:32.257931 23651 hybrid_clock.cc:648] HybridClock initialized: now 1790516972257868 us; error 59 us; skew 500 ppm
I20260927 13:49:32.258407 23651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:32 dist-test-slave-q25r krb5kdc[23411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516972, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:32.269392 23651 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:49:32.271138 23651 webserver.cc:460] Webserver started at http://127.20.133.66:42495/ using document root <none> and password file <none>
I20260927 13:49:32.271910 23651 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:32.272043 23651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:32.272413 23651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:32.275569 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "a727225821f84538a8b5de9c8ba514d0"
format_stamp: "Formatted at 2026-09-27 13:49:32 on dist-test-slave-q25r"
server_key: "4a14a087992db6af6e2485cb34b30e03"
server_key_iv: "53db16403049bb5afc9d97201a4451fe"
server_key_version: "encryptionkey@0"
I20260927 13:49:32.276495 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "a727225821f84538a8b5de9c8ba514d0"
format_stamp: "Formatted at 2026-09-27 13:49:32 on dist-test-slave-q25r"
server_key: "4a14a087992db6af6e2485cb34b30e03"
server_key_iv: "53db16403049bb5afc9d97201a4451fe"
server_key_version: "encryptionkey@0"
I20260927 13:49:32.282353 23651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:49:32.286237 23666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:32.287667 23651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:49:32.287842 23651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "a727225821f84538a8b5de9c8ba514d0"
format_stamp: "Formatted at 2026-09-27 13:49:32 on dist-test-slave-q25r"
server_key: "4a14a087992db6af6e2485cb34b30e03"
server_key_iv: "53db16403049bb5afc9d97201a4451fe"
server_key_version: "encryptionkey@0"
I20260927 13:49:32.288053 23651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:32.317698 23651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:32.321981 23651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:32.322270 23651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:32.323246 23651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:32.324697 23651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:32.324779 23651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:32.324910 23651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:32.324966 23651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:32.355943 23651 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:38175
I20260927 13:49:32.355993 23778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:38175 every 8 connection(s)
I20260927 13:49:32.357299 23651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:49:32.363766 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 23651
I20260927 13:49:32.364009 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:49:32.364579 21013 external_mini_cluster.cc:1428] Setting key 603e8aadb3079c85440eafe11e992429
Sep 27 13:49:32 dist-test-slave-q25r krb5kdc[23411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516972, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:32.381413 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:32.359989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:46217 (local address 127.20.133.126:32913)
0927 13:49:32.360476 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:32.360482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:32.361561 (+ 1079us) server_negotiation.cc:407] Connection header received
0927 13:49:32.363481 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:32.363485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:32.363562 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:32.363668 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:32.365678 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:32.366719 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:32.369195 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:32.369687 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:32.369809 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:32.374549 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:32.374557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:32.374562 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:32.374629 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:32.377891 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:32.378811 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:32.378817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:32.378821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:32.378989 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:32.379426 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:32.379431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:32.379434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:32.379815 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:32.379971 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:32.380773 (+ 802us) server_negotiation.cc:299] Negotiation successful
0927 13:49:32.381034 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":224,"threads_started":1}
I20260927 13:49:32.382375 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:32.360621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:32913 (local address 127.20.133.66:46217)
0927 13:49:32.361361 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:32.361400 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:32.363152 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:32.364005 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:32.364019 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:32.364822 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:32.365540 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:32.365558 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:32.366875 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:32.366882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:32.368983 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:32.369003 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:32.369950 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:32.369956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:32.370266 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:32.371078 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:32.371123 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:32.374176 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:32.378120 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:32.378130 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:32.378146 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:32.378660 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:32.379099 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:32.379102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:32.379104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:32.379311 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:32.379930 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:32.379936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:32.380294 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 13:49:32.380619 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 13:49:32.381148 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":266,"threads_started":1}
I20260927 13:49:32.384262 23779 heartbeater.cc:347] Connected to a master server at 127.20.133.126:32913
I20260927 13:49:32.384601 23779 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
I20260927 13:49:32.385397 23779 heartbeater.cc:511] Master 127.20.133.126:32913 requested a full tablet report, sending...
I20260927 13:49:32.387130 23456 ts_manager.cc:194] Registered new tserver with Master: a727225821f84538a8b5de9c8ba514d0 (127.20.133.66:38175)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
I20260927 13:49:32.388058 23456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:46217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:49:32.411072 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32913
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:32.734932 23785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:32.735217 23785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:32.735312 23785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:32.742832 23785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:32.742961 23785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:32.743076 23785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:49:32.753244 23785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:32.754498 23785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:32.755721 23785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:32.768689 23793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:32.768788 23791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:32.768757 23790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:32.769811 23785 server_base.cc:1034] running on GCE node
I20260927 13:49:32.770427 23785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:32.771442 23785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:32.772646 23785 hybrid_clock.cc:648] HybridClock initialized: now 1790516972772592 us; error 52 us; skew 500 ppm
I20260927 13:49:32.772979 23785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:32 dist-test-slave-q25r krb5kdc[23411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516972, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:32.776068 23785 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:49:32.777603 23785 webserver.cc:460] Webserver started at http://127.20.133.67:45663/ using document root <none> and password file <none>
I20260927 13:49:32.778344 23785 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:32.778456 23785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:32.778741 23785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:32.781613 23785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "739df63d69e34b409109f976930552f6"
format_stamp: "Formatted at 2026-09-27 13:49:32 on dist-test-slave-q25r"
server_key: "fba92025b42786b6a7c3b0dce3d26d1c"
server_key_iv: "515a72b59e9141d8797db033d402afab"
server_key_version: "encryptionkey@0"
I20260927 13:49:32.782373 23785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "739df63d69e34b409109f976930552f6"
format_stamp: "Formatted at 2026-09-27 13:49:32 on dist-test-slave-q25r"
server_key: "fba92025b42786b6a7c3b0dce3d26d1c"
server_key_iv: "515a72b59e9141d8797db033d402afab"
server_key_version: "encryptionkey@0"
I20260927 13:49:32.787550 23785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 13:49:32.791473 23800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:32.793049 23785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:49:32.793197 23785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "739df63d69e34b409109f976930552f6"
format_stamp: "Formatted at 2026-09-27 13:49:32 on dist-test-slave-q25r"
server_key: "fba92025b42786b6a7c3b0dce3d26d1c"
server_key_iv: "515a72b59e9141d8797db033d402afab"
server_key_version: "encryptionkey@0"
I20260927 13:49:32.793387 23785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:32.821718 23785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:32.826449 23785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:32.826710 23785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:32.827641 23785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:32.829025 23785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:32.829098 23785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:32.829191 23785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:32.829234 23785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:32.852161 23645 heartbeater.cc:503] Master 127.20.133.126:32913 was elected leader, sending a full tablet report...
I20260927 13:49:32.860888 23785 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:45655
I20260927 13:49:32.860967 23912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:45655 every 8 connection(s)
I20260927 13:49:32.862200 23785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:49:32.869949 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 23785
I20260927 13:49:32.870169 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:49:32.870761 21013 external_mini_cluster.cc:1428] Setting key d1830a0f9e0dac9c8de99af6c9f84736
Sep 27 13:49:32 dist-test-slave-q25r krb5kdc[23411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516972, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:32.883041 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:32.865024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:58943 (local address 127.20.133.126:32913)
0927 13:49:32.865248 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:32.865256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:32.866477 (+ 1221us) server_negotiation.cc:407] Connection header received
0927 13:49:32.868323 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:32.868328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:32.868480 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:32.868593 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:32.870114 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:32.871259 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:32.873200 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:32.873633 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:32.873730 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:32.877368 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:32.877374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:32.877377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:32.877410 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:32.880073 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:32.880918 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:32.880923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:32.880925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:32.881082 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:32.881484 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:32.881489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:32.881491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:32.881810 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:32.881943 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:32.882632 (+ 689us) server_negotiation.cc:299] Negotiation successful
0927 13:49:32.882815 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 13:49:32.884076 23915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:32.865538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:32913 (local address 127.20.133.67:58943)
0927 13:49:32.866309 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:32.866349 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:32.868027 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:32.868792 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:32.868800 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:32.869327 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:32.869914 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:32.869932 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:32.871427 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:32.871433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:32.873069 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:32.873084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:32.873869 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:32.873877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:32.874171 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:32.874910 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:32.874934 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:32.877138 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:32.880244 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:32.880251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:32.880266 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:32.880783 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:32.881187 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:32.881190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:32.881192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:32.881384 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:32.881915 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:32.881922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:32.882248 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 13:49:32.882476 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 13:49:32.882951 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":288,"threads_started":1}
I20260927 13:49:32.885928 23913 heartbeater.cc:347] Connected to a master server at 127.20.133.126:32913
I20260927 13:49:32.886263 23913 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:32.887061 23913 heartbeater.cc:511] Master 127.20.133.126:32913 requested a full tablet report, sending...
I20260927 13:49:32.888648 23455 ts_manager.cc:194] Registered new tserver with Master: 739df63d69e34b409109f976930552f6 (127.20.133.67:45655)
I20260927 13:49:32.889616 23455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:58943
I20260927 13:49:32.899441 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 13:49:32.929921 23781 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:49:32.927229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.20.133.126:32913)
0927 13:49:32.927412 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:32.927417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:32.927876 (+ 459us) server_negotiation.cc:407] Connection header received
0927 13:49:32.928068 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:32.928073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:32.928184 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:32.928279 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:32.929694 (+ 1415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38020: BlockingRecv error: recv got EOF from 127.0.0.1:38020 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:49:32.933375 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 23517
W20260927 13:49:32.936229 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 23517: Not authorized: Client connection negotiation failed: client connection to 127.20.133.65:35695: server requires authentication, but client does not have Kerberos credentials available
I20260927 13:49:32.936436 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 23517
I20260927 13:49:32.949366 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 23651
W20260927 13:49:32.951972 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 23651: Not authorized: Client connection negotiation failed: client connection to 127.20.133.66:38175: server requires authentication, but client does not have Kerberos credentials available
I20260927 13:49:32.952191 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 23651
I20260927 13:49:32.964439 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 23785
W20260927 13:49:32.967497 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 23785: Not authorized: Client connection negotiation failed: client connection to 127.20.133.67:45655: server requires authentication, but client does not have Kerberos credentials available
I20260927 13:49:32.967615 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 23785
I20260927 13:49:32.979399 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 23424
I20260927 13:49:32.980695 23486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:33.087070 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 23424
2026-09-27T13:49:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3630 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:49:33 dist-test-slave-q25r krb5kdc[23930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:49:33 dist-test-slave-q25r krb5kdc[23930](info): set up 2 sockets
Sep 27 13:49:33 dist-test-slave-q25r krb5kdc[23930](info): commencing operation
krb5kdc: starting...
W20260927 13:49:34.104936 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:49:34 dist-test-slave-q25r krb5kdc[23930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:49:34.323945 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:37691
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:38119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:37691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:34.657902 23943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:34.658231 23943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:34.658324 23943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:34.666539 23943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:34.666667 23943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:34.666740 23943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:34.666800 23943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:34.666847 23943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:34.676964 23943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:38119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:37691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:37691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:34.678313 23943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:34.679831 23943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:34.691819 23948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:34.691856 23951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:34.691883 23949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:34.692536 23943 server_base.cc:1034] running on GCE node
I20260927 13:49:34.693397 23943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:34.694939 23943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:34.696189 23943 hybrid_clock.cc:648] HybridClock initialized: now 1790516974696131 us; error 51 us; skew 500 ppm
I20260927 13:49:34.696632 23943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:34 dist-test-slave-q25r krb5kdc[23930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516974, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:34.699965 23943 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:49:34.701612 23943 webserver.cc:460] Webserver started at http://127.20.133.126:34045/ using document root <none> and password file <none>
I20260927 13:49:34.702332 23943 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:34.702453 23943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:34.702760 23943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:34.705974 23943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "d0593785b5d54c5ab29f23c03236ffd4"
format_stamp: "Formatted at 2026-09-27 13:49:34 on dist-test-slave-q25r"
server_key: "f870c318fe126151b9c344f4eadfe9b4"
server_key_iv: "acf48ae9d1f2b1ff5f858973ef3dfe5f"
server_key_version: "encryptionkey@0"
I20260927 13:49:34.706784 23943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "d0593785b5d54c5ab29f23c03236ffd4"
format_stamp: "Formatted at 2026-09-27 13:49:34 on dist-test-slave-q25r"
server_key: "f870c318fe126151b9c344f4eadfe9b4"
server_key_iv: "acf48ae9d1f2b1ff5f858973ef3dfe5f"
server_key_version: "encryptionkey@0"
I20260927 13:49:34.712280 23943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 13:49:34.716207 23958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:34.717756 23943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:49:34.717934 23943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "d0593785b5d54c5ab29f23c03236ffd4"
format_stamp: "Formatted at 2026-09-27 13:49:34 on dist-test-slave-q25r"
server_key: "f870c318fe126151b9c344f4eadfe9b4"
server_key_iv: "acf48ae9d1f2b1ff5f858973ef3dfe5f"
server_key_version: "encryptionkey@0"
I20260927 13:49:34.718132 23943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:34.741729 23943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:34.746102 23943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:34.746392 23943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:34.764037 23943 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:37691
I20260927 13:49:34.764073 24009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:37691 every 8 connection(s)
I20260927 13:49:34.765372 23943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:34.769032 24010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:34.775188 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 23943
I20260927 13:49:34.775411 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:34.776003 21013 external_mini_cluster.cc:1428] Setting key d25ae932d4384b7b93e96edec0f5c39e
I20260927 13:49:34.779934 24010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4: Bootstrap starting.
I20260927 13:49:34.783793 24010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:34.785194 24010 log.cc:824] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:34.787961 24010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4: No bootstrap required, opened a new log
Sep 27 13:49:34 dist-test-slave-q25r krb5kdc[23930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:34.795723 24010 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0593785b5d54c5ab29f23c03236ffd4" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37691 } }
I20260927 13:49:34.796135 24010 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:34.796265 24010 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0593785b5d54c5ab29f23c03236ffd4, State: Initialized, Role: FOLLOWER
I20260927 13:49:34.796885 24010 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0593785b5d54c5ab29f23c03236ffd4" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37691 } }
I20260927 13:49:34.797127 24010 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:34.797298 24010 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:34.797441 24010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:34.799283 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:34.778036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.20.133.126:37691)
0927 13:49:34.778856 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:34.778871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:34.778920 (+ 49us) server_negotiation.cc:407] Connection header received
0927 13:49:34.781366 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:34.781397 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:34.781967 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:34.782446 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:34.783919 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:34.785939 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:34.787228 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:34.787884 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:34.788057 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:34.791746 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:34.791754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:34.791770 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:34.791813 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:34.795003 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:34.795761 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:34.795769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:34.795777 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:34.795950 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:34.796393 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:34.796398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:34.796402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:34.797030 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:34.797317 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:34.797796 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 13:49:34.798239 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":354,"threads_started":1}
I20260927 13:49:34.799913 24010 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0593785b5d54c5ab29f23c03236ffd4" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37691 } }
I20260927 13:49:34.800467 24010 leader_election.cc:302] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0593785b5d54c5ab29f23c03236ffd4; no voters:
I20260927 13:49:34.801148 24010 leader_election.cc:288] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:34.801317 24015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:34.802402 24015 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [term 1 LEADER]: Becoming Leader. State: Replica: d0593785b5d54c5ab29f23c03236ffd4, State: Running, Role: LEADER
I20260927 13:49:34.803027 24015 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0593785b5d54c5ab29f23c03236ffd4" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37691 } }
I20260927 13:49:34.803363 24010 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:34.803871 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:49:34.806136 24016 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0593785b5d54c5ab29f23c03236ffd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0593785b5d54c5ab29f23c03236ffd4" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37691 } } }
I20260927 13:49:34.806545 24016 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:34.807191 24017 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0593785b5d54c5ab29f23c03236ffd4. Latest consensus state: current_term: 1 leader_uuid: "d0593785b5d54c5ab29f23c03236ffd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0593785b5d54c5ab29f23c03236ffd4" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37691 } } }
I20260927 13:49:34.807579 24017 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:34.812457 24021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:34.819566 24021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:34.830474 24021 catalog_manager.cc:1347] Generated new cluster ID: 51f21ee3ab3d47fba0c3c090789e8dcd
I20260927 13:49:34.830591 24021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:34.853343 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:34.843047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.20.133.126:37691)
0927 13:49:34.843249 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:34.843257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:34.844933 (+ 1676us) server_negotiation.cc:407] Connection header received
0927 13:49:34.845117 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:34.845123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:34.845208 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:34.845320 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:34.846570 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:34.847658 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:34.849524 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:34.850221 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:34.850354 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:34.851604 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:34.851610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:34.851615 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:34.852499 (+ 884us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:34.852580 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:34.852606 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:34.852727 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:34.852853 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 13:49:34.852958 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":47}
I20260927 13:49:34.853797 24021 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:34.855391 24021 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:34.865836 24021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0593785b5d54c5ab29f23c03236ffd4: Generated new TSK 0
I20260927 13:49:34.866775 24021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:49:34.879827 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 23943
I20260927 13:49:34.881244 24005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:34.973014 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 23943
2026-09-27T13:49:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1879 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:49:35 dist-test-slave-q25r krb5kdc[24044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:49:35 dist-test-slave-q25r krb5kdc[24044](info): set up 2 sockets
Sep 27 13:49:35 dist-test-slave-q25r krb5kdc[24044](info): commencing operation
krb5kdc: starting...
W20260927 13:49:35.968856 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:49:36 dist-test-slave-q25r krb5kdc[24044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:49:36.168421 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:38599
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:38599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:36.492000 24057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:36.492390 24057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:36.492723 24057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:36.500237 24057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:36.500419 24057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:36.500506 24057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:36.500586 24057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:36.500645 24057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:36.510929 24057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:38599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:38599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:36.512290 24057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:36.513610 24057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:36.524446 24062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:36.524483 24065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:36.524501 24063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:36.525642 24057 server_base.cc:1034] running on GCE node
I20260927 13:49:36.526377 24057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:36.527706 24057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:36.528901 24057 hybrid_clock.cc:648] HybridClock initialized: now 1790516976528860 us; error 30 us; skew 500 ppm
I20260927 13:49:36.529219 24057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:36 dist-test-slave-q25r krb5kdc[24044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516976, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:36.532222 24057 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:49:36.533728 24057 webserver.cc:460] Webserver started at http://127.20.133.126:43403/ using document root <none> and password file <none>
I20260927 13:49:36.534387 24057 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:36.534494 24057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:36.534798 24057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:36.537837 24057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "ada9b2ec2b6d4eac91212df6f094f8c0"
format_stamp: "Formatted at 2026-09-27 13:49:36 on dist-test-slave-q25r"
server_key: "284b2599205f14945e02d6747d482b33"
server_key_iv: "91fe2169bd41cd88ea5b8370ac0e2788"
server_key_version: "encryptionkey@0"
I20260927 13:49:36.538614 24057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "ada9b2ec2b6d4eac91212df6f094f8c0"
format_stamp: "Formatted at 2026-09-27 13:49:36 on dist-test-slave-q25r"
server_key: "284b2599205f14945e02d6747d482b33"
server_key_iv: "91fe2169bd41cd88ea5b8370ac0e2788"
server_key_version: "encryptionkey@0"
I20260927 13:49:36.543799 24057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 13:49:36.547101 24072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:36.548393 24057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:49:36.548552 24057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "ada9b2ec2b6d4eac91212df6f094f8c0"
format_stamp: "Formatted at 2026-09-27 13:49:36 on dist-test-slave-q25r"
server_key: "284b2599205f14945e02d6747d482b33"
server_key_iv: "91fe2169bd41cd88ea5b8370ac0e2788"
server_key_version: "encryptionkey@0"
I20260927 13:49:36.548725 24057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:36.562544 24057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:36.567000 24057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:36.567282 24057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:36.585686 24057 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:38599
I20260927 13:49:36.585698 24123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:38599 every 8 connection(s)
I20260927 13:49:36.587123 24057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:36.590981 24124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:36.597332 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 24057
I20260927 13:49:36.597571 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:36.598430 21013 external_mini_cluster.cc:1428] Setting key 02610fb30a753ebe7428fc5e57620119
I20260927 13:49:36.603266 24124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0: Bootstrap starting.
I20260927 13:49:36.606515 24124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:36.607534 24124 log.cc:824] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:36.610324 24124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0: No bootstrap required, opened a new log
Sep 27 13:49:36 dist-test-slave-q25r krb5kdc[24044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:36.617856 24124 raft_consensus.cc:348] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada9b2ec2b6d4eac91212df6f094f8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38599 } }
I20260927 13:49:36.618252 24124 raft_consensus.cc:374] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:36.618363 24124 raft_consensus.cc:729] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada9b2ec2b6d4eac91212df6f094f8c0, State: Initialized, Role: FOLLOWER
I20260927 13:49:36.618980 24124 consensus_queue.cc:260] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada9b2ec2b6d4eac91212df6f094f8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38599 } }
I20260927 13:49:36.619251 24124 raft_consensus.cc:388] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:36.619376 24124 raft_consensus.cc:482] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:36.619547 24124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:36.620442 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:36.601005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.20.133.126:38599)
0927 13:49:36.601850 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:36.601862 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:36.601906 (+ 44us) server_negotiation.cc:407] Connection header received
0927 13:49:36.603693 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:36.603724 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:36.604124 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:36.604511 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:36.605939 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:36.607237 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:36.608623 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:36.609277 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:36.609441 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:36.613201 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:36.613209 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:36.613227 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:36.613269 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:36.616256 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:36.616962 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:36.616969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:36.616976 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:36.617152 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:36.617518 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:36.617522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:36.617524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:36.618090 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:36.618403 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:36.618839 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 13:49:36.619281 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":398,"threads_started":1}
I20260927 13:49:36.622058 24124 raft_consensus.cc:504] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada9b2ec2b6d4eac91212df6f094f8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38599 } }
I20260927 13:49:36.622540 24124 leader_election.cc:302] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ada9b2ec2b6d4eac91212df6f094f8c0; no voters:
I20260927 13:49:36.623157 24124 leader_election.cc:288] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:36.623282 24129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:36.623924 24129 raft_consensus.cc:686] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [term 1 LEADER]: Becoming Leader. State: Replica: ada9b2ec2b6d4eac91212df6f094f8c0, State: Running, Role: LEADER
I20260927 13:49:36.624465 24129 consensus_queue.cc:237] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada9b2ec2b6d4eac91212df6f094f8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38599 } }
I20260927 13:49:36.625327 24124 sys_catalog.cc:564] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:36.626979 24130 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ada9b2ec2b6d4eac91212df6f094f8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada9b2ec2b6d4eac91212df6f094f8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38599 } } }
I20260927 13:49:36.627288 24130 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:36.628074 24135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:36.627907 24131 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ada9b2ec2b6d4eac91212df6f094f8c0. Latest consensus state: current_term: 1 leader_uuid: "ada9b2ec2b6d4eac91212df6f094f8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada9b2ec2b6d4eac91212df6f094f8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38599 } } }
I20260927 13:49:36.628821 24131 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:36.633814 24135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:36.644060 24135 catalog_manager.cc:1347] Generated new cluster ID: 48051d24c2324e558d1fb7046db66473
I20260927 13:49:36.644179 24135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:36.669968 24135 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:36.671563 24135 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:36.681710 24135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ada9b2ec2b6d4eac91212df6f094f8c0: Generated new TSK 0
I20260927 13:49:36.682801 24135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:49:36.736275 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38599
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:37.057339 24150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:37.057641 24150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:37.057781 24150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:37.065505 24150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:37.065655 24150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:37.065822 24150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:49:37.075888 24150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:37.077181 24150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:37.078423 24150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:37.091216 24158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:37.091178 24156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:37.091248 24155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:37.091969 24150 server_base.cc:1034] running on GCE node
I20260927 13:49:37.092664 24150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:37.093654 24150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:37.094880 24150 hybrid_clock.cc:648] HybridClock initialized: now 1790516977094815 us; error 58 us; skew 500 ppm
I20260927 13:49:37.095305 24150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:37 dist-test-slave-q25r krb5kdc[24044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516977, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:37.098487 24150 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:49:37.100167 24150 webserver.cc:460] Webserver started at http://127.20.133.65:33817/ using document root <none> and password file <none>
I20260927 13:49:37.100942 24150 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:37.101054 24150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:37.101305 24150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:37.104413 24150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "89ca464933ac4354905f06d0c468ebec"
format_stamp: "Formatted at 2026-09-27 13:49:37 on dist-test-slave-q25r"
server_key: "c49df1ffc7a62d56995916ea88517624"
server_key_iv: "2f17808e6f6bd39455c2e893e9be97b0"
server_key_version: "encryptionkey@0"
I20260927 13:49:37.105302 24150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "89ca464933ac4354905f06d0c468ebec"
format_stamp: "Formatted at 2026-09-27 13:49:37 on dist-test-slave-q25r"
server_key: "c49df1ffc7a62d56995916ea88517624"
server_key_iv: "2f17808e6f6bd39455c2e893e9be97b0"
server_key_version: "encryptionkey@0"
I20260927 13:49:37.110670 24150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 13:49:37.114223 24165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:37.115612 24150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:49:37.115763 24150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "89ca464933ac4354905f06d0c468ebec"
format_stamp: "Formatted at 2026-09-27 13:49:37 on dist-test-slave-q25r"
server_key: "c49df1ffc7a62d56995916ea88517624"
server_key_iv: "2f17808e6f6bd39455c2e893e9be97b0"
server_key_version: "encryptionkey@0"
I20260927 13:49:37.115949 24150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:37.168147 24150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:37.172569 24150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:37.172842 24150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:37.173784 24150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:37.175140 24150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:37.175215 24150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:37.175314 24150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:37.175357 24150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:37.205886 24150 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:35845
I20260927 13:49:37.205929 24277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:35845 every 8 connection(s)
I20260927 13:49:37.207118 24150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:49:37.217134 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 24150
I20260927 13:49:37.217351 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:49:37.218020 21013 external_mini_cluster.cc:1428] Setting key eeb7dbd5ed8c077cb3733cc0a27b5c0e
Sep 27 13:49:37 dist-test-slave-q25r krb5kdc[24044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516977, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:37.227154 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:37.209698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:58401 (local address 127.20.133.126:38599)
0927 13:49:37.210132 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:37.210136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:37.210926 (+ 790us) server_negotiation.cc:407] Connection header received
0927 13:49:37.212691 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:37.212696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:37.212778 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:37.212873 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:37.214306 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:37.215031 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:37.216609 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:37.217217 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:37.217346 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:37.221130 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:37.221136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:37.221142 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:37.221182 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:37.224091 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:37.224865 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:37.224870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:37.224873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:37.225007 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:37.225410 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:37.225414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:37.225417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:37.225735 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:37.225873 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:37.226705 (+ 832us) server_negotiation.cc:299] Negotiation successful
0927 13:49:37.226877 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":216,"threads_started":1}
I20260927 13:49:37.228525 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:37.210109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38599 (local address 127.20.133.65:58401)
0927 13:49:37.210774 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:37.210814 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:37.212441 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:37.213103 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:37.213111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:37.213601 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:37.214144 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:37.214161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:37.215170 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:37.215174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:37.216471 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:37.216483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:37.217480 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:37.217488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:37.217695 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:37.218319 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:37.218346 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:37.220882 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:37.224247 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:37.224254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:37.224268 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:37.224749 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:37.225099 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:37.225104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:37.225107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:37.225313 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:37.225858 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:37.225865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:37.226309 (+ 444us) client_negotiation.cc:769] Sending connection context
0927 13:49:37.226618 (+ 309us) client_negotiation.cc:240] Negotiation successful
0927 13:49:37.227174 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":251,"threads_started":1}
I20260927 13:49:37.230147 24278 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38599
I20260927 13:49:37.230472 24278 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:37.231256 24278 heartbeater.cc:511] Master 127.20.133.126:38599 requested a full tablet report, sending...
I20260927 13:49:37.233592 24089 ts_manager.cc:194] Registered new tserver with Master: 89ca464933ac4354905f06d0c468ebec (127.20.133.65:35845)
I20260927 13:49:37.235121 24089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:58401
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:49:37.263481 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38599
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:37.576148 24284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:37.576464 24284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:37.576567 24284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:37.583825 24284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:37.583949 24284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:37.584059 24284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:49:37.598869 24284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:37.600440 24284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:37.602048 24284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:37.615091 24289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:37.615054 24292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:37.615063 24290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:37.615510 24284 server_base.cc:1034] running on GCE node
I20260927 13:49:37.616185 24284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:37.617084 24284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:37.618287 24284 hybrid_clock.cc:648] HybridClock initialized: now 1790516977618236 us; error 36 us; skew 500 ppm
I20260927 13:49:37.618556 24284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:37 dist-test-slave-q25r krb5kdc[24044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516977, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:37.624758 24284 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:49:37.626174 24284 webserver.cc:460] Webserver started at http://127.20.133.66:38891/ using document root <none> and password file <none>
I20260927 13:49:37.626827 24284 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:37.626925 24284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:37.627202 24284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:37.630060 24284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "54a01d4230bd4782ab6c081e2b6ca551"
format_stamp: "Formatted at 2026-09-27 13:49:37 on dist-test-slave-q25r"
server_key: "be4f785df843965e4cc5523ecf7236c9"
server_key_iv: "83015431bddf226a618d12cc51acd79d"
server_key_version: "encryptionkey@0"
I20260927 13:49:37.630821 24284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "54a01d4230bd4782ab6c081e2b6ca551"
format_stamp: "Formatted at 2026-09-27 13:49:37 on dist-test-slave-q25r"
server_key: "be4f785df843965e4cc5523ecf7236c9"
server_key_iv: "83015431bddf226a618d12cc51acd79d"
server_key_version: "encryptionkey@0"
I20260927 13:49:37.635861 24284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:49:37.639226 24299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:37.640575 24284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:49:37.640724 24284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "54a01d4230bd4782ab6c081e2b6ca551"
format_stamp: "Formatted at 2026-09-27 13:49:37 on dist-test-slave-q25r"
server_key: "be4f785df843965e4cc5523ecf7236c9"
server_key_iv: "83015431bddf226a618d12cc51acd79d"
server_key_version: "encryptionkey@0"
I20260927 13:49:37.640904 24284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:37.672870 24284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:37.677099 24284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:37.677352 24284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:37.678294 24284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:37.679581 24284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:37.679653 24284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:37.679741 24284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:37.679783 24284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:37.708913 24284 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:43705
I20260927 13:49:37.708982 24411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:43705 every 8 connection(s)
I20260927 13:49:37.710145 24284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:49:37.711731 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 24284
I20260927 13:49:37.711876 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:49:37.712440 21013 external_mini_cluster.cc:1428] Setting key 94655277d269bc7466ef7814e5581ce3
Sep 27 13:49:37 dist-test-slave-q25r krb5kdc[24044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516977, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
I20260927 13:49:37.730168 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:37.712698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:47697 (local address 127.20.133.126:38599)
0927 13:49:37.712866 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:37.712870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:37.714273 (+ 1403us) server_negotiation.cc:407] Connection header received
0927 13:49:37.716101 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:37.716107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:37.716197 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:37.716281 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:37.717737 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:37.718428 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:37.720486 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:37.720891 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:37.721018 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:37.724500 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:37.724507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:37.724512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:37.724545 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:37.727021 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:37.727755 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:37.727759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:37.727761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:37.727885 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:37.728295 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:37.728298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:37.728300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:37.728602 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:37.728746 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:37.729487 (+ 741us) server_negotiation.cc:299] Negotiation successful
0927 13:49:37.729834 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:49:37.731086 24414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:37.713307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38599 (local address 127.20.133.66:47697)
0927 13:49:37.714117 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:37.714164 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:37.715848 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:37.716537 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:37.716545 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:37.717040 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:37.717571 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:37.717588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:37.718572 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:37.718575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:37.720330 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:37.720372 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:37.721124 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:37.721132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:37.721349 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:37.721959 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:37.721986 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:37.724252 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:37.727166 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:37.727176 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:37.727195 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:37.727630 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:37.727986 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:37.727991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:37.727995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:37.728183 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:37.728724 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:37.728732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:37.729098 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 13:49:37.729362 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 13:49:37.729901 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":326,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
I20260927 13:49:37.732566 24412 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38599
I20260927 13:49:37.732923 24412 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:37.733791 24412 heartbeater.cc:511] Master 127.20.133.126:38599 requested a full tablet report, sending...
I20260927 13:49:37.735275 24089 ts_manager.cc:194] Registered new tserver with Master: 54a01d4230bd4782ab6c081e2b6ca551 (127.20.133.66:43705)
I20260927 13:49:37.736140 24089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:47697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:49:37.755364 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38599
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:38.072005 24417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:38.072443 24417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:38.072576 24417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:38.080185 24417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:38.080312 24417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:38.080497 24417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:49:38.090942 24417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:38.092381 24417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:38.093739 24417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:38.106580 24425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:38.106736 24423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:38.106817 24422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:38.107759 24417 server_base.cc:1034] running on GCE node
I20260927 13:49:38.108408 24417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:38.109416 24417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:38.110673 24417 hybrid_clock.cc:648] HybridClock initialized: now 1790516978110614 us; error 46 us; skew 500 ppm
I20260927 13:49:38.111047 24417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:38 dist-test-slave-q25r krb5kdc[24044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516978, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:38.114543 24417 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:49:38.116078 24417 webserver.cc:460] Webserver started at http://127.20.133.67:36343/ using document root <none> and password file <none>
I20260927 13:49:38.116827 24417 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:38.116950 24417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:38.117355 24417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:38.120488 24417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "028cca023c6b45abb3944f707ae2eb22"
format_stamp: "Formatted at 2026-09-27 13:49:38 on dist-test-slave-q25r"
server_key: "bb94949d14eced87e19bb8341b81d301"
server_key_iv: "52f374182ed1a19c20a8f60aea79e812"
server_key_version: "encryptionkey@0"
I20260927 13:49:38.121289 24417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "028cca023c6b45abb3944f707ae2eb22"
format_stamp: "Formatted at 2026-09-27 13:49:38 on dist-test-slave-q25r"
server_key: "bb94949d14eced87e19bb8341b81d301"
server_key_iv: "52f374182ed1a19c20a8f60aea79e812"
server_key_version: "encryptionkey@0"
I20260927 13:49:38.126768 24417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 13:49:38.130327 24432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:38.131705 24417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:49:38.131891 24417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "028cca023c6b45abb3944f707ae2eb22"
format_stamp: "Formatted at 2026-09-27 13:49:38 on dist-test-slave-q25r"
server_key: "bb94949d14eced87e19bb8341b81d301"
server_key_iv: "52f374182ed1a19c20a8f60aea79e812"
server_key_version: "encryptionkey@0"
I20260927 13:49:38.132090 24417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:38.145159 24417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:38.149350 24417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:38.149626 24417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:38.150609 24417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:38.152009 24417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:38.152084 24417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:38.152204 24417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:38.152253 24417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:38.181317 24417 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:36851
I20260927 13:49:38.181353 24544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:36851 every 8 connection(s)
I20260927 13:49:38.182570 24417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:49:38.183259 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 24417
I20260927 13:49:38.183458 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:49:38.184041 21013 external_mini_cluster.cc:1428] Setting key 91bebeb73ec6c7adcbb1921e31abf92b
Sep 27 13:49:38 dist-test-slave-q25r krb5kdc[24044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516978, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:38.203855 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:38.185301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:39159 (local address 127.20.133.126:38599)
0927 13:49:38.185470 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:38.185476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:38.186845 (+ 1369us) server_negotiation.cc:407] Connection header received
0927 13:49:38.188830 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:38.188834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:38.188913 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:38.189007 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:38.190854 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.191696 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.193521 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.193934 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.194043 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:38.197675 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:38.197681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:38.197684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:38.197732 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:38.200795 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:38.201547 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:38.201551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:38.201553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:38.201685 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:38.202122 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:38.202125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:38.202127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:38.202395 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:38.202532 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:38.203248 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 13:49:38.203486 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 13:49:38.204835 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:38.185853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38599 (local address 127.20.133.67:39159)
0927 13:49:38.186698 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:38.186744 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:38.188535 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:38.189224 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:38.189233 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:38.189805 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:38.190578 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:38.190607 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.191901 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.191906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:38.193335 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:38.193356 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.194178 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.194184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:38.194414 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:38.195077 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:38.195101 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:38.197426 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:38.200941 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:38.200948 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:38.200964 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:38.201420 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:38.201790 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:38.201795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:38.201798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:38.201998 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:38.202523 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:38.202532 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:38.202876 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 13:49:38.203117 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 13:49:38.203698 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":344,"threads_started":1}
I20260927 13:49:38.206225 24545 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38599
I20260927 13:49:38.206521 24545 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:38.207229 24545 heartbeater.cc:511] Master 127.20.133.126:38599 requested a full tablet report, sending...
I20260927 13:49:38.208702 24089 ts_manager.cc:194] Registered new tserver with Master: 028cca023c6b45abb3944f707ae2eb22 (127.20.133.67:36851)
I20260927 13:49:38.209563 24089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:39159
I20260927 13:49:38.211941 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:49:38 dist-test-slave-q25r krb5kdc[24044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516978, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260927 13:49:38.238662 24278 heartbeater.cc:503] Master 127.20.133.126:38599 was elected leader, sending a full tablet report...
Sep 27 13:49:38 dist-test-slave-q25r krb5kdc[24044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516978, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:38.260124 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:38.245141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.20.133.126:38599)
0927 13:49:38.245332 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:38.245338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:38.245906 (+ 568us) server_negotiation.cc:407] Connection header received
0927 13:49:38.246101 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:38.246107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:38.246201 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:38.246301 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:38.248013 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.248841 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.250160 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.250605 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.250733 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:38.254850 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:38.254855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:38.254859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:38.254897 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:38.257530 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:38.258207 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:38.258213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:38.258216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:38.258348 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:38.258718 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:38.258723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:38.258727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:38.259065 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:38.259301 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:38.259574 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 13:49:38.259790 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 13:49:38.260766 24089 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41454
I20260927 13:49:38.262740 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 24150
Sep 27 13:49:38 dist-test-slave-q25r krb5kdc[24044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516978, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:38.277837 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:38.263607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.20.133.65:35845)
0927 13:49:38.264189 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:38.264196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:38.264220 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:49:38.264275 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:38.264282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:38.264524 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:38.264666 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:38.265834 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.266850 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.268038 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.268702 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.268818 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:38.272456 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:38.272468 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:38.272481 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:38.272514 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:38.275158 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:38.275807 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:38.275815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:38.275817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:38.275949 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:38.276253 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:38.276256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:38.276258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:38.276657 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:38.276900 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:38.277224 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 13:49:38.277491 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":248,"threads_started":1}
W20260927 13:49:38.279126 24273 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51650
W20260927 13:49:38.280303 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 24150: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:49:38.280541 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 24150
I20260927 13:49:38.292789 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 24284
Sep 27 13:49:38 dist-test-slave-q25r krb5kdc[24044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516978, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:49:38.309341 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:38.293694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.20.133.66:43705)
0927 13:49:38.294230 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:38.294237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:38.294263 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:49:38.294332 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:38.294338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:38.294544 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:38.294674 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:38.295778 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.296854 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.298126 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.298712 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.298814 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:38.303070 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:38.303086 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:38.303100 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:38.303146 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:38.306321 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:38.307070 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:38.307078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:38.307081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:38.307246 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:38.307660 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:38.307664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:38.307666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:38.308023 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:38.308249 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:38.308697 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 13:49:38.308931 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":288,"threads_started":1}
W20260927 13:49:38.310567 24407 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36664
W20260927 13:49:38.311587 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 24284: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:49:38.311743 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 24284
I20260927 13:49:38.324061 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 24417
Sep 27 13:49:38 dist-test-slave-q25r krb5kdc[24044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516978, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:49:38.340317 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:38.325033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.20.133.67:36851)
0927 13:49:38.325712 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:38.325723 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:38.325762 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:49:38.325828 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:38.325835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:38.326066 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:38.326196 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:38.327619 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.328784 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.330066 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:38.330643 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:38.330752 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:38.334941 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:38.334954 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:38.334971 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:38.335009 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:38.337799 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:38.338421 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:38.338427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:38.338430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:38.338539 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:38.338889 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:38.338892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:38.338894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:38.339220 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:38.339465 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:38.339754 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 13:49:38.339955 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":315,"threads_started":1}
W20260927 13:49:38.341583 24540 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48778
W20260927 13:49:38.342581 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 24417: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:49:38.342725 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 24417
I20260927 13:49:38.354629 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 24057
I20260927 13:49:38.355705 24119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:38.457228 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 24057
2026-09-27T13:49:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3490 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:49:38 dist-test-slave-q25r krb5kdc[24562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:49:38 dist-test-slave-q25r krb5kdc[24562](info): set up 2 sockets
Sep 27 13:49:38 dist-test-slave-q25r krb5kdc[24562](info): commencing operation
krb5kdc: starting...
W20260927 13:49:39.456784 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:49:39 dist-test-slave-q25r krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:49:39.680016 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:45483
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:45483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:40.002339 24575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:40.002657 24575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:40.002768 24575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:40.010182 24575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:40.010309 24575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:40.010354 24575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:40.010435 24575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:40.010499 24575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:40.020586 24575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:45483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:45483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:40.021932 24575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:40.023495 24575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:40.036198 24580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:40.036445 24583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:40.036484 24581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:40.037657 24575 server_base.cc:1034] running on GCE node
I20260927 13:49:40.038496 24575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:40.040036 24575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:40.041297 24575 hybrid_clock.cc:648] HybridClock initialized: now 1790516980041217 us; error 68 us; skew 500 ppm
I20260927 13:49:40.041708 24575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:40 dist-test-slave-q25r krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516980, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:40.045014 24575 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:49:40.046612 24575 webserver.cc:460] Webserver started at http://127.20.133.126:38745/ using document root <none> and password file <none>
I20260927 13:49:40.047304 24575 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:40.047420 24575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:40.047716 24575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:40.050638 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "80041f81fbfb407d8747298d43c8d8c0"
format_stamp: "Formatted at 2026-09-27 13:49:40 on dist-test-slave-q25r"
server_key: "313defe304053b1a1f6491a8c11e15cc"
server_key_iv: "c2b61cf517bb4b856a5207b5744e9ea9"
server_key_version: "encryptionkey@0"
I20260927 13:49:40.051383 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "80041f81fbfb407d8747298d43c8d8c0"
format_stamp: "Formatted at 2026-09-27 13:49:40 on dist-test-slave-q25r"
server_key: "313defe304053b1a1f6491a8c11e15cc"
server_key_iv: "c2b61cf517bb4b856a5207b5744e9ea9"
server_key_version: "encryptionkey@0"
I20260927 13:49:40.056761 24575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 13:49:40.060628 24590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:40.062479 24575 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:49:40.062714 24575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "80041f81fbfb407d8747298d43c8d8c0"
format_stamp: "Formatted at 2026-09-27 13:49:40 on dist-test-slave-q25r"
server_key: "313defe304053b1a1f6491a8c11e15cc"
server_key_iv: "c2b61cf517bb4b856a5207b5744e9ea9"
server_key_version: "encryptionkey@0"
I20260927 13:49:40.062929 24575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:40.076700 24575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:40.081355 24575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:40.081671 24575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:40.101994 24575 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:45483
I20260927 13:49:40.102048 24641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:45483 every 8 connection(s)
I20260927 13:49:40.103551 24575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:40.108258 24642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:40.111835 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 24575
I20260927 13:49:40.112069 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:40.112789 21013 external_mini_cluster.cc:1428] Setting key 1b17c5c92e2f1130354ebb82eb343fe6
I20260927 13:49:40.123255 24642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0: Bootstrap starting.
I20260927 13:49:40.127538 24642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:40.128759 24642 log.cc:824] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:49:40 dist-test-slave-q25r krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:40.132795 24642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0: No bootstrap required, opened a new log
I20260927 13:49:40.139178 24645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:40.115161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.20.133.126:45483)
0927 13:49:40.116128 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:40.116147 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:40.116212 (+ 65us) server_negotiation.cc:407] Connection header received
0927 13:49:40.118493 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:40.118526 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:40.119199 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:40.119731 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:40.121178 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:40.123210 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:40.124591 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:40.125728 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:40.125947 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:40.129964 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:40.129973 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:40.129990 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:40.130044 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:40.134079 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:40.134856 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:40.134872 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:40.134881 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:40.135118 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:40.135483 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:40.135488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:40.135492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:40.136319 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:40.136687 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:40.137135 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 13:49:40.137655 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":369,"threads_started":1}
I20260927 13:49:40.143836 24642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80041f81fbfb407d8747298d43c8d8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45483 } }
I20260927 13:49:40.144238 24642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:40.144323 24642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80041f81fbfb407d8747298d43c8d8c0, State: Initialized, Role: FOLLOWER
I20260927 13:49:40.145005 24642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80041f81fbfb407d8747298d43c8d8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45483 } }
I20260927 13:49:40.145241 24642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:40.145344 24642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:40.145495 24642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:40.148138 24642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80041f81fbfb407d8747298d43c8d8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45483 } }
I20260927 13:49:40.148739 24642 leader_election.cc:302] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80041f81fbfb407d8747298d43c8d8c0; no voters:
I20260927 13:49:40.149402 24642 leader_election.cc:288] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:40.149605 24647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:40.150388 24647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [term 1 LEADER]: Becoming Leader. State: Replica: 80041f81fbfb407d8747298d43c8d8c0, State: Running, Role: LEADER
I20260927 13:49:40.151098 24647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80041f81fbfb407d8747298d43c8d8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45483 } }
I20260927 13:49:40.151933 24642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:40.155190 24648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80041f81fbfb407d8747298d43c8d8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80041f81fbfb407d8747298d43c8d8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45483 } } }
I20260927 13:49:40.155751 24648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:40.156536 24649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80041f81fbfb407d8747298d43c8d8c0. Latest consensus state: current_term: 1 leader_uuid: "80041f81fbfb407d8747298d43c8d8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80041f81fbfb407d8747298d43c8d8c0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45483 } } }
I20260927 13:49:40.156921 24649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:40.159093 24655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:40.164230 24655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:40.175115 24655 catalog_manager.cc:1347] Generated new cluster ID: 5d4d85bc6f594dceab92b625aee0b95b
I20260927 13:49:40.175259 24655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:40.203153 24655 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:40.204746 24655 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:40.211403 24655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80041f81fbfb407d8747298d43c8d8c0: Generated new TSK 0
I20260927 13:49:40.212913 24655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:49:40.266544 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45483
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:40.607771 24668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:40.608111 24668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:40.608250 24668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:40.616122 24668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:40.616279 24668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:40.616436 24668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:49:40.626453 24668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:40.627797 24668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:40.629156 24668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:40.642426 24673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:40.642390 24676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:40.642359 24674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:40.642882 24668 server_base.cc:1034] running on GCE node
I20260927 13:49:40.643675 24668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:40.644726 24668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:40.645944 24668 hybrid_clock.cc:648] HybridClock initialized: now 1790516980645903 us; error 40 us; skew 500 ppm
I20260927 13:49:40.646277 24668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:40 dist-test-slave-q25r krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516980, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:40.649493 24668 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:49:40.650982 24668 webserver.cc:460] Webserver started at http://127.20.133.65:37441/ using document root <none> and password file <none>
I20260927 13:49:40.651656 24668 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:40.651753 24668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:40.652011 24668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:40.655092 24668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "c9d67ed9d42d46aaa1b848f3f9010335"
format_stamp: "Formatted at 2026-09-27 13:49:40 on dist-test-slave-q25r"
server_key: "5b9bd3eb773f3d4882b8bd9e5eafd6bc"
server_key_iv: "0c0e08049a9e2cf3dd69134f1a07eaee"
server_key_version: "encryptionkey@0"
I20260927 13:49:40.655838 24668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "c9d67ed9d42d46aaa1b848f3f9010335"
format_stamp: "Formatted at 2026-09-27 13:49:40 on dist-test-slave-q25r"
server_key: "5b9bd3eb773f3d4882b8bd9e5eafd6bc"
server_key_iv: "0c0e08049a9e2cf3dd69134f1a07eaee"
server_key_version: "encryptionkey@0"
I20260927 13:49:40.661360 24668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 13:49:40.665055 24683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:40.666450 24668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:49:40.666594 24668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "c9d67ed9d42d46aaa1b848f3f9010335"
format_stamp: "Formatted at 2026-09-27 13:49:40 on dist-test-slave-q25r"
server_key: "5b9bd3eb773f3d4882b8bd9e5eafd6bc"
server_key_iv: "0c0e08049a9e2cf3dd69134f1a07eaee"
server_key_version: "encryptionkey@0"
I20260927 13:49:40.666796 24668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:40.700855 24668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:40.704895 24668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:40.705180 24668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:40.706142 24668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:40.707506 24668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:40.707578 24668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:40.707697 24668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:40.707754 24668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:40.738220 24668 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:42185
I20260927 13:49:40.738269 24795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:42185 every 8 connection(s)
I20260927 13:49:40.739503 24668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:49:40.741504 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 24668
I20260927 13:49:40.741706 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:49:40.742478 21013 external_mini_cluster.cc:1428] Setting key 71b1f9c15d151762a89297b47485fc96
Sep 27 13:49:40 dist-test-slave-q25r krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516980, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
I20260927 13:49:40.762548 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:40.742096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:58233 (local address 127.20.133.126:45483)
0927 13:49:40.742636 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:40.742642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:40.743785 (+ 1143us) server_negotiation.cc:407] Connection header received
0927 13:49:40.746245 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:40.746250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:40.746323 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:40.746421 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:40.748575 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:40.749319 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:40.751557 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:40.752192 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:40.752379 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:40.756297 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:40.756304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:40.756308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:40.756379 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:40.759251 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:40.760106 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:40.760111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:40.760113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:40.760253 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:40.760733 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:40.760738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:40.760741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:40.761016 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:40.761152 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:40.761931 (+ 779us) server_negotiation.cc:299] Negotiation successful
0927 13:49:40.762168 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":273,"threads_started":1}
I20260927 13:49:40.763625 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:40.742784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45483 (local address 127.20.133.65:58233)
0927 13:49:40.743617 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:40.743681 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:40.745906 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:40.746736 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:40.746748 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:40.747486 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:40.748306 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:40.748366 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:40.749493 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:40.749498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:40.751354 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:40.751374 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:40.752523 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:40.752530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:40.752775 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:40.753485 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:40.753529 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:40.755994 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:40.759427 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:40.759437 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:40.759454 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:40.759962 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:40.760391 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:40.760396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:40.760399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:40.760609 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:40.761148 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:40.761157 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:40.761529 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 13:49:40.761796 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 13:49:40.762360 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":313,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
I20260927 13:49:40.765314 24796 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45483
I20260927 13:49:40.765612 24796 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:40.766407 24796 heartbeater.cc:511] Master 127.20.133.126:45483 requested a full tablet report, sending...
I20260927 13:49:40.768808 24607 ts_manager.cc:194] Registered new tserver with Master: c9d67ed9d42d46aaa1b848f3f9010335 (127.20.133.65:42185)
I20260927 13:49:40.770665 24607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:58233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:49:40.789165 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45483
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:41.123939 24802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:41.124270 24802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:41.124434 24802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:41.131860 24802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:41.132002 24802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:41.132160 24802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:49:41.142669 24802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:41.144074 24802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:41.145457 24802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:41.158502 24808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:41.158531 24807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:41.159065 24810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:41.159739 24802 server_base.cc:1034] running on GCE node
I20260927 13:49:41.160365 24802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:41.161376 24802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:41.162596 24802 hybrid_clock.cc:648] HybridClock initialized: now 1790516981162533 us; error 59 us; skew 500 ppm
I20260927 13:49:41.162923 24802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:41 dist-test-slave-q25r krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516981, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:41.166209 24802 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:49:41.167919 24802 webserver.cc:460] Webserver started at http://127.20.133.66:44893/ using document root <none> and password file <none>
I20260927 13:49:41.168646 24802 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:41.168751 24802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:41.169013 24802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:41.172061 24802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "109e4691ecd04a5f949cd36fa55ede17"
format_stamp: "Formatted at 2026-09-27 13:49:41 on dist-test-slave-q25r"
server_key: "18d81bfb1414fe400d894033507a2368"
server_key_iv: "56fe73329db5e5344b62084f63f32962"
server_key_version: "encryptionkey@0"
I20260927 13:49:41.172869 24802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "109e4691ecd04a5f949cd36fa55ede17"
format_stamp: "Formatted at 2026-09-27 13:49:41 on dist-test-slave-q25r"
server_key: "18d81bfb1414fe400d894033507a2368"
server_key_iv: "56fe73329db5e5344b62084f63f32962"
server_key_version: "encryptionkey@0"
I20260927 13:49:41.177961 24802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:49:41.181293 24817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:41.182600 24802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:49:41.182766 24802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "109e4691ecd04a5f949cd36fa55ede17"
format_stamp: "Formatted at 2026-09-27 13:49:41 on dist-test-slave-q25r"
server_key: "18d81bfb1414fe400d894033507a2368"
server_key_iv: "56fe73329db5e5344b62084f63f32962"
server_key_version: "encryptionkey@0"
I20260927 13:49:41.182956 24802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:41.210980 24802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:41.216188 24802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:41.216527 24802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:41.217549 24802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:41.219038 24802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:41.219115 24802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:41.219216 24802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:41.219270 24802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:41.250351 24802 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:37599
I20260927 13:49:41.250386 24929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:37599 every 8 connection(s)
I20260927 13:49:41.251545 24802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:49:41.258853 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 24802
I20260927 13:49:41.259022 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:49:41.259471 21013 external_mini_cluster.cc:1428] Setting key 32f231d13e3ed46a27a36a197a500942
Sep 27 13:49:41 dist-test-slave-q25r krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516981, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:41.273154 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:41.254129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:40585 (local address 127.20.133.126:45483)
0927 13:49:41.254332 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:41.254338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:41.255493 (+ 1155us) server_negotiation.cc:407] Connection header received
0927 13:49:41.257487 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:41.257492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:41.257565 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:41.257652 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:41.259410 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.260414 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.262263 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.262883 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.263013 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:41.266901 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:41.266909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:41.266914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:41.266960 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:41.269562 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.270583 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.270588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.270591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.270750 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.271283 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.271288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.271290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.271655 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:41.271768 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:41.272690 (+ 922us) server_negotiation.cc:299] Negotiation successful
0927 13:49:41.272900 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:49:41.274519 24932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:41.254613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45483 (local address 127.20.133.66:40585)
0927 13:49:41.255330 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:41.255366 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:41.257191 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:41.257900 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:41.257908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:41.258560 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:41.259226 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:41.259246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.260628 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.260634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:41.262104 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:41.262123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.263109 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.263114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:41.263344 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:41.264030 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:41.264104 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:41.266625 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:41.269747 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:41.269757 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:41.269778 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:41.270394 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:41.270864 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:41.270868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:41.270873 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:41.271124 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:41.271771 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:41.271779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:41.272238 (+ 459us) client_negotiation.cc:769] Sending connection context
0927 13:49:41.272560 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 13:49:41.273133 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":288,"threads_started":1}
I20260927 13:49:41.276108 24930 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45483
I20260927 13:49:41.276484 24930 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:41.277386 24930 heartbeater.cc:511] Master 127.20.133.126:45483 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
I20260927 13:49:41.279011 24607 ts_manager.cc:194] Registered new tserver with Master: 109e4691ecd04a5f949cd36fa55ede17 (127.20.133.66:37599)
I20260927 13:49:41.279831 24607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:40585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:49:41.303102 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45483
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:41.625787 24935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:41.626075 24935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:41.626166 24935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:41.633405 24935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:41.633533 24935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:41.633656 24935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:49:41.643575 24935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:41.644905 24935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:41.646195 24935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:41.660414 24940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:41.660719 24941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:41.661226 24943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:41.661835 24935 server_base.cc:1034] running on GCE node
I20260927 13:49:41.662503 24935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:41.663611 24935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:41.664848 24935 hybrid_clock.cc:648] HybridClock initialized: now 1790516981664782 us; error 58 us; skew 500 ppm
I20260927 13:49:41.665216 24935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:41 dist-test-slave-q25r krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516981, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:41.668876 24935 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:49:41.670675 24935 webserver.cc:460] Webserver started at http://127.20.133.67:46271/ using document root <none> and password file <none>
I20260927 13:49:41.671437 24935 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:41.671567 24935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:41.671865 24935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:41.674845 24935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "26a982a6b02e42c6a9f6a0728b96f2f7"
format_stamp: "Formatted at 2026-09-27 13:49:41 on dist-test-slave-q25r"
server_key: "b3c12ce77333e0d51b90290b7a01c21d"
server_key_iv: "87bbadfdaf9d28a68be7bb5e2cc2d97f"
server_key_version: "encryptionkey@0"
I20260927 13:49:41.675618 24935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "26a982a6b02e42c6a9f6a0728b96f2f7"
format_stamp: "Formatted at 2026-09-27 13:49:41 on dist-test-slave-q25r"
server_key: "b3c12ce77333e0d51b90290b7a01c21d"
server_key_iv: "87bbadfdaf9d28a68be7bb5e2cc2d97f"
server_key_version: "encryptionkey@0"
I20260927 13:49:41.681362 24935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 13:49:41.685357 24950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:41.686980 24935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:49:41.687160 24935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "26a982a6b02e42c6a9f6a0728b96f2f7"
format_stamp: "Formatted at 2026-09-27 13:49:41 on dist-test-slave-q25r"
server_key: "b3c12ce77333e0d51b90290b7a01c21d"
server_key_iv: "87bbadfdaf9d28a68be7bb5e2cc2d97f"
server_key_version: "encryptionkey@0"
I20260927 13:49:41.687395 24935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:41.703008 24935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:41.707477 24935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:41.707773 24935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:41.708873 24935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:41.710281 24935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:41.710358 24935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:41.710474 24935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:41.710530 24935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:41.745098 24935 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:43045
I20260927 13:49:41.745113 25062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:43045 every 8 connection(s)
I20260927 13:49:41.746420 24935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:49:41.752530 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 24935
I20260927 13:49:41.752741 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:49:41.753244 21013 external_mini_cluster.cc:1428] Setting key 99eb06cd5919caff31ba0321502be837
Sep 27 13:49:41 dist-test-slave-q25r krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516981, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:41.769977 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:41.749563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:56667 (local address 127.20.133.126:45483)
0927 13:49:41.749800 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:41.749805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:41.751221 (+ 1416us) server_negotiation.cc:407] Connection header received
0927 13:49:41.753800 (+ 2579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:41.753807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:41.753954 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:41.754067 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:41.755863 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.757090 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.758889 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.759315 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.759430 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:41.763260 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:41.763266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:41.763270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:41.763360 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:41.766164 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.767020 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.767027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.767029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.767178 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.767651 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.767657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.767660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.768048 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:41.768220 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:41.769288 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0927 13:49:41.769602 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 13:49:41.771000 25065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:41.750103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45483 (local address 127.20.133.67:56667)
0927 13:49:41.751005 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:41.751062 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:41.753383 (+ 2321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:41.754359 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:41.754369 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:41.754980 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:41.755683 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:41.755702 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.757268 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.757272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:41.758724 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:41.758739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.759543 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.759549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:41.759797 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:41.760572 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:41.760599 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:41.763023 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:41.766329 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:41.766337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:41.766351 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:41.766875 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:41.767292 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:41.767296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:41.767299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:41.767517 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:41.768225 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:41.768234 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:41.768810 (+ 576us) client_negotiation.cc:769] Sending connection context
0927 13:49:41.769136 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 13:49:41.769727 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":357,"threads_started":1}
I20260927 13:49:41.773038 25063 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45483
I20260927 13:49:41.773430 25063 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:41.774358 24796 heartbeater.cc:503] Master 127.20.133.126:45483 was elected leader, sending a full tablet report...
I20260927 13:49:41.774463 25063 heartbeater.cc:511] Master 127.20.133.126:45483 requested a full tablet report, sending...
I20260927 13:49:41.776216 24607 ts_manager.cc:194] Registered new tserver with Master: 26a982a6b02e42c6a9f6a0728b96f2f7 (127.20.133.67:43045)
I20260927 13:49:41.777082 24607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:56667
I20260927 13:49:41.782922 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:49:41 dist-test-slave-q25r krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:49:41 dist-test-slave-q25r krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:41.838447 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:41.821556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.20.133.65:42185)
0927 13:49:41.822342 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:41.822353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:41.822388 (+ 35us) server_negotiation.cc:407] Connection header received
0927 13:49:41.822489 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:41.822497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:41.822789 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:41.822966 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:41.824673 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.825813 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.827092 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.827722 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.827846 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:41.832214 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:41.832227 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:41.832247 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:41.832293 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:41.835498 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.836195 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.836204 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.836206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.836382 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.836741 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.836747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.836750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.837158 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:41.837491 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:41.837761 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 13:49:41.838072 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":352,"threads_started":1}
Sep 27 13:49:41 dist-test-slave-q25r krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:41.858973 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:41.843656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.20.133.126:45483)
0927 13:49:41.843953 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:41.843959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:41.844112 (+ 153us) server_negotiation.cc:407] Connection header received
0927 13:49:41.844285 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:41.844291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:41.844407 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:41.844511 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:41.845800 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.846600 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.847938 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.848413 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.848528 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:41.853274 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:41.853281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:41.853285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:41.853343 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:41.856001 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.856883 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.856888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.856891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.857061 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.857499 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.857503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.857505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.857941 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:41.858161 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:41.858517 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 13:49:41.858698 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260927 13:49:41.859751 24607 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51230
I20260927 13:49:41.861055 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 24668
I20260927 13:49:41.875182 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:41.862022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.20.133.65:42185)
0927 13:49:41.862203 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:41.862208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:41.862432 (+ 224us) server_negotiation.cc:407] Connection header received
0927 13:49:41.862600 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:41.862605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:41.862699 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:41.862789 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:41.864145 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.865092 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.866454 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.866932 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.867041 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:41.869580 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:41.869586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:41.869589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:41.869633 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:41.872625 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.873319 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.873326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.873330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.873502 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.873870 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.873874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.873876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.874208 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:41.874438 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:41.874663 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 13:49:41.874902 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:49:41.875968 24791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:41.967044 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 24668
I20260927 13:49:41.982708 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 24802
Sep 27 13:49:41 dist-test-slave-q25r krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:49:42.000000 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:41.983842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.20.133.66:37599)
0927 13:49:41.984567 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:41.984575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:41.984604 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:49:41.984664 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:41.984671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:41.984918 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:41.985071 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:41.986359 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.987410 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.988835 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:41.989398 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:41.989525 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:41.993905 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:41.993925 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:41.993944 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:41.993981 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:41.996840 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.997524 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.997531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.997534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.997693 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:41.998297 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:41.998302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:41.998304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:41.998720 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:41.998997 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:41.999349 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 13:49:41.999591 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":306,"threads_started":1}
I20260927 13:49:42.001891 24925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:42.095490 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 24802
I20260927 13:49:42.110821 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 24935
Sep 27 13:49:42 dist-test-slave-q25r krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:49:42.130478 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:42.112110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.20.133.67:43045)
0927 13:49:42.113015 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:42.113023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:42.113057 (+ 34us) server_negotiation.cc:407] Connection header received
0927 13:49:42.113128 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:42.113135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:42.113358 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:42.113510 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:42.114981 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:42.116073 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:42.117548 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:42.118182 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:42.118306 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:42.123430 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:42.123451 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:42.123473 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:42.123531 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:42.126797 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:42.127744 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:42.127752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:42.127755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:42.127933 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:42.128629 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:42.128632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:42.128635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:42.129089 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:42.129399 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:42.129841 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 13:49:42.130114 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":525,"threads_started":1}
I20260927 13:49:42.132222 25058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:42.218338 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 24935
I20260927 13:49:42.232079 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 24575
I20260927 13:49:42.233541 24637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:42.347127 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 24575
2026-09-27T13:49:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3891 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:49:42 dist-test-slave-q25r krb5kdc[25082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:49:42 dist-test-slave-q25r krb5kdc[25082](info): set up 2 sockets
Sep 27 13:49:42 dist-test-slave-q25r krb5kdc[25082](info): commencing operation
krb5kdc: starting...
W20260927 13:49:43.368877 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:49:43 dist-test-slave-q25r krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:49:43.592119 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:32969
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:43.931599 25095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:43.931919 25095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:43.932058 25095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:43.939683 25095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:43.939832 25095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:43.939918 25095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:43.939999 25095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:43.940068 25095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:43.950223 25095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:43.951551 25095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:43.952881 25095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:43.964111 25100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:43.964167 25103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:43.964191 25101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:43.965193 25095 server_base.cc:1034] running on GCE node
I20260927 13:49:43.966024 25095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:43.967489 25095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:43.968705 25095 hybrid_clock.cc:648] HybridClock initialized: now 1790516983968642 us; error 51 us; skew 500 ppm
I20260927 13:49:43.969080 25095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:43 dist-test-slave-q25r krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516983, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:43.972278 25095 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:49:43.973507 25095 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:43.973623 25095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:43.973937 25095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:43.976995 25095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "119a28e1d84c4ceca198c6d0e354f6bb"
format_stamp: "Formatted at 2026-09-27 13:49:43 on dist-test-slave-q25r"
server_key: "4b47b0b82dee2774e6006400a98678ed"
server_key_iv: "2bf2b73432a9047e3f154522bf0b3c7b"
server_key_version: "encryptionkey@0"
I20260927 13:49:43.977813 25095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "119a28e1d84c4ceca198c6d0e354f6bb"
format_stamp: "Formatted at 2026-09-27 13:49:43 on dist-test-slave-q25r"
server_key: "4b47b0b82dee2774e6006400a98678ed"
server_key_iv: "2bf2b73432a9047e3f154522bf0b3c7b"
server_key_version: "encryptionkey@0"
I20260927 13:49:43.982921 25095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:49:43.986531 25109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:43.988006 25095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 13:49:43.988209 25095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "119a28e1d84c4ceca198c6d0e354f6bb"
format_stamp: "Formatted at 2026-09-27 13:49:43 on dist-test-slave-q25r"
server_key: "4b47b0b82dee2774e6006400a98678ed"
server_key_iv: "2bf2b73432a9047e3f154522bf0b3c7b"
server_key_version: "encryptionkey@0"
I20260927 13:49:43.988497 25095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:44.012466 25095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:44.017001 25095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:44.017273 25095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:44.035388 25095 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:32969
I20260927 13:49:44.035408 25160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:32969 every 8 connection(s)
I20260927 13:49:44.036271 25095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:44.040844 25161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:44.041808 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 25095
I20260927 13:49:44.042073 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:44.042896 21013 external_mini_cluster.cc:1428] Setting key 616d9a9207c40d5ecc2a4e2a83ac52c7
I20260927 13:49:44.055733 25161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb: Bootstrap starting.
I20260927 13:49:44.059504 25161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:44.060616 25161 log.cc:824] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:49:44 dist-test-slave-q25r krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:44.064113 25161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb: No bootstrap required, opened a new log
I20260927 13:49:44.070890 25164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:44.045393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.20.133.126:32969)
0927 13:49:44.046360 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:44.046377 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:44.049016 (+ 2639us) server_negotiation.cc:407] Connection header received
0927 13:49:44.050997 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:44.051030 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:44.051609 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:44.052128 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:44.053645 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:44.055728 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:44.057363 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:44.058429 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:44.058626 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:44.063160 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:44.063169 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:44.063186 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:44.063226 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:44.066452 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:44.067377 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:44.067384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:44.067392 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:44.067579 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:44.068019 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:44.068023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:44.068025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:44.068681 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:44.068998 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:44.069391 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 13:49:44.069836 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":397,"threads_started":1}
I20260927 13:49:44.073014 25161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a28e1d84c4ceca198c6d0e354f6bb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32969 } }
I20260927 13:49:44.073365 25161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:44.073509 25161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119a28e1d84c4ceca198c6d0e354f6bb, State: Initialized, Role: FOLLOWER
I20260927 13:49:44.074092 25161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a28e1d84c4ceca198c6d0e354f6bb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32969 } }
I20260927 13:49:44.074295 25161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:44.074388 25161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:44.074505 25161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:44.076633 25161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a28e1d84c4ceca198c6d0e354f6bb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32969 } }
I20260927 13:49:44.077090 25161 leader_election.cc:302] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 119a28e1d84c4ceca198c6d0e354f6bb; no voters:
I20260927 13:49:44.077628 25161 leader_election.cc:288] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:44.077785 25166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:44.078387 25166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [term 1 LEADER]: Becoming Leader. State: Replica: 119a28e1d84c4ceca198c6d0e354f6bb, State: Running, Role: LEADER
I20260927 13:49:44.078946 25166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a28e1d84c4ceca198c6d0e354f6bb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32969 } }
I20260927 13:49:44.079694 25161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:44.081713 25168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 119a28e1d84c4ceca198c6d0e354f6bb. Latest consensus state: current_term: 1 leader_uuid: "119a28e1d84c4ceca198c6d0e354f6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a28e1d84c4ceca198c6d0e354f6bb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32969 } } }
I20260927 13:49:44.082114 25168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:44.081916 25167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "119a28e1d84c4ceca198c6d0e354f6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a28e1d84c4ceca198c6d0e354f6bb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 32969 } } }
I20260927 13:49:44.082228 25167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:44.083964 25174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:44.088712 25174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:44.097205 25174 catalog_manager.cc:1347] Generated new cluster ID: 7082976a13f94b2eb4769c503450e947
I20260927 13:49:44.097339 25174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:44.117594 25174 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:44.119295 25174 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:44.134822 25174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb: Generated new TSK 0
I20260927 13:49:44.135720 25174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:49:44.186131 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32969
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:44.557723 25187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:44.558020 25187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:44.558163 25187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:44.565706 25187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:44.565838 25187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:44.566004 25187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:49:44.576665 25187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:44.578009 25187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:44.579326 25187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:44.593538 25195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:44.593490 25192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:44.593462 25193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:44.593786 25187 server_base.cc:1034] running on GCE node
I20260927 13:49:44.594702 25187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:44.595750 25187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:44.596972 25187 hybrid_clock.cc:648] HybridClock initialized: now 1790516984596917 us; error 51 us; skew 500 ppm
I20260927 13:49:44.597301 25187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:44 dist-test-slave-q25r krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516984, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:44.600497 25187 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:49:44.601636 25187 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:44.601725 25187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:44.601989 25187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:44.604964 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "e4eb0593068e42b58f676755d96232f4"
format_stamp: "Formatted at 2026-09-27 13:49:44 on dist-test-slave-q25r"
server_key: "0eb770aebbd82bc8cd98e8df9d7e7069"
server_key_iv: "c82ee17f661dc6868603b0d008e632ac"
server_key_version: "encryptionkey@0"
I20260927 13:49:44.605739 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "e4eb0593068e42b58f676755d96232f4"
format_stamp: "Formatted at 2026-09-27 13:49:44 on dist-test-slave-q25r"
server_key: "0eb770aebbd82bc8cd98e8df9d7e7069"
server_key_iv: "c82ee17f661dc6868603b0d008e632ac"
server_key_version: "encryptionkey@0"
I20260927 13:49:44.610837 25187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:49:44.614405 25201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:44.615712 25187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:49:44.615864 25187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "e4eb0593068e42b58f676755d96232f4"
format_stamp: "Formatted at 2026-09-27 13:49:44 on dist-test-slave-q25r"
server_key: "0eb770aebbd82bc8cd98e8df9d7e7069"
server_key_iv: "c82ee17f661dc6868603b0d008e632ac"
server_key_version: "encryptionkey@0"
I20260927 13:49:44.616089 25187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:44.641230 25187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:44.645689 25187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:44.645961 25187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:44.646957 25187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:44.648301 25187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:44.648423 25187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:44.648530 25187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:44.648604 25187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:44.679719 25187 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:43949
I20260927 13:49:44.679816 25313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:43949 every 8 connection(s)
I20260927 13:49:44.680713 25187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:49:44.681182 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 25187
I20260927 13:49:44.681394 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:49:44.682164 21013 external_mini_cluster.cc:1428] Setting key 249d5a8491f201e2e7b2c2f5b7545a43
Sep 27 13:49:44 dist-test-slave-q25r krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516984, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
I20260927 13:49:44.701030 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:44.683240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:33137 (local address 127.20.133.126:32969)
0927 13:49:44.683694 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:44.683698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:44.684775 (+ 1077us) server_negotiation.cc:407] Connection header received
0927 13:49:44.686582 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:44.686587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:44.686669 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:44.686760 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:44.688202 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:44.688997 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:44.691026 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:44.691431 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:44.691546 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:44.695197 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:44.695203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:44.695206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:44.695241 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:44.697822 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:44.698607 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:44.698614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:44.698617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:44.698741 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:44.699172 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:44.699176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:44.699178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:44.699467 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:44.699599 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:44.700396 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 13:49:44.700632 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
I20260927 13:49:44.702066 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:44.683873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:32969 (local address 127.20.133.65:33137)
0927 13:49:44.684631 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:44.684685 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:44.686339 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:44.686981 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:44.686989 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:44.687483 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:44.688038 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:44.688055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:44.689148 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:44.689152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:44.690888 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:44.690902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:44.691668 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:44.691676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:44.691901 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:44.692652 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:44.692678 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:44.694941 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:44.698002 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:44.698011 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:44.698027 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:44.698485 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:44.698850 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:44.698855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:44.698859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:44.699056 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:44.699593 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:44.699611 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:44.699970 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 13:49:44.700246 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 13:49:44.700855 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":309,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
I20260927 13:49:44.703882 25314 heartbeater.cc:347] Connected to a master server at 127.20.133.126:32969
I20260927 13:49:44.704171 25314 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:44.704878 25314 heartbeater.cc:511] Master 127.20.133.126:32969 requested a full tablet report, sending...
I20260927 13:49:44.707202 25126 ts_manager.cc:194] Registered new tserver with Master: e4eb0593068e42b58f676755d96232f4 (127.20.133.65:43949)
I20260927 13:49:44.708948 25126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:33137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:49:44.727679 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32969
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:45.057549 25320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:45.057862 25320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:45.057961 25320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:45.065486 25320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:45.065613 25320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:45.065734 25320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:49:45.076419 25320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:45.077667 25320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:45.078866 25320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:45.091853 25326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:45.091876 25325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:45.092237 25328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:45.092880 25320 server_base.cc:1034] running on GCE node
I20260927 13:49:45.093562 25320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:45.094568 25320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:45.095763 25320 hybrid_clock.cc:648] HybridClock initialized: now 1790516985095707 us; error 51 us; skew 500 ppm
I20260927 13:49:45.096067 25320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:45 dist-test-slave-q25r krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516985, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:45.099202 25320 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:49:45.100426 25320 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:45.100524 25320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:45.100829 25320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:45.103719 25320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "2941c414a60b45cfba7b5df6ce7783ce"
format_stamp: "Formatted at 2026-09-27 13:49:45 on dist-test-slave-q25r"
server_key: "ca146b312c201d3b1d3faeb23aafb03d"
server_key_iv: "070948adcd2aa62ee880c5363769b115"
server_key_version: "encryptionkey@0"
I20260927 13:49:45.104550 25320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "2941c414a60b45cfba7b5df6ce7783ce"
format_stamp: "Formatted at 2026-09-27 13:49:45 on dist-test-slave-q25r"
server_key: "ca146b312c201d3b1d3faeb23aafb03d"
server_key_iv: "070948adcd2aa62ee880c5363769b115"
server_key_version: "encryptionkey@0"
I20260927 13:49:45.109463 25320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 13:49:45.112938 25334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:45.114387 25320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:49:45.114554 25320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "2941c414a60b45cfba7b5df6ce7783ce"
format_stamp: "Formatted at 2026-09-27 13:49:45 on dist-test-slave-q25r"
server_key: "ca146b312c201d3b1d3faeb23aafb03d"
server_key_iv: "070948adcd2aa62ee880c5363769b115"
server_key_version: "encryptionkey@0"
I20260927 13:49:45.114770 25320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:45.127429 25320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:45.131734 25320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:45.132005 25320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:45.133029 25320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:45.134402 25320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:45.134496 25320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:45.134601 25320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:45.134670 25320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:45.166162 25320 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:33531
I20260927 13:49:45.166211 25446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:33531 every 8 connection(s)
I20260927 13:49:45.167027 25320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:49:45.177250 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 25320
I20260927 13:49:45.177469 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:49:45.177996 21013 external_mini_cluster.cc:1428] Setting key e03e411b060a37113715849810859a17
Sep 27 13:49:45 dist-test-slave-q25r krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516985, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:45.186721 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:45.169439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:57479 (local address 127.20.133.126:32969)
0927 13:49:45.169646 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:45.169653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:45.170809 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 13:49:45.172638 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:45.172644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:45.172723 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:45.172817 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:45.174340 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.175035 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.176615 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.177105 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.177219 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:45.181040 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:45.181046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:45.181049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:45.181080 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:45.183562 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:45.184304 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:45.184310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:45.184314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:45.184485 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:45.184913 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:45.184918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:45.184921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:45.185187 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:45.185339 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:45.186136 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 13:49:45.186385 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:49:45.187714 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:45.169961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:32969 (local address 127.20.133.66:57479)
0927 13:49:45.170637 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:45.170673 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:45.172300 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:45.173020 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:45.173029 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:45.173541 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:45.174137 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:45.174154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.175167 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.175170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:45.176459 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:45.176472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.177393 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.177402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:45.177706 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:45.178406 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:45.178442 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:45.180826 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:45.183704 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:45.183715 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:45.183734 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:45.184168 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:45.184594 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:45.184599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:45.184603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:45.184798 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:45.185318 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:45.185326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:45.185763 (+ 437us) client_negotiation.cc:769] Sending connection context
0927 13:49:45.186010 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 13:49:45.186554 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":265,"threads_started":1}
I20260927 13:49:45.189162 25447 heartbeater.cc:347] Connected to a master server at 127.20.133.126:32969
I20260927 13:49:45.189482 25447 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:45.190230 25447 heartbeater.cc:511] Master 127.20.133.126:32969 requested a full tablet report, sending...
I20260927 13:49:45.191785 25126 ts_manager.cc:194] Registered new tserver with Master: 2941c414a60b45cfba7b5df6ce7783ce (127.20.133.66:33531)
I20260927 13:49:45.192700 25126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:57479
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:49:45.225003 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32969
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:45.554350 25452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:45.554670 25452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:45.554800 25452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:45.562148 25452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:45.562269 25452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:45.562459 25452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:49:45.575240 25452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:32969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:45.576647 25452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:45.577932 25452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:45.590936 25457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:45.590869 25458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:45.590912 25460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:45.591727 25452 server_base.cc:1034] running on GCE node
I20260927 13:49:45.592324 25452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:45.593333 25452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:45.594604 25452 hybrid_clock.cc:648] HybridClock initialized: now 1790516985594538 us; error 48 us; skew 500 ppm
I20260927 13:49:45.594938 25452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:45 dist-test-slave-q25r krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516985, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:45.598061 25452 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:49:45.599255 25452 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:45.599361 25452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:45.599684 25452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:45.602759 25452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "e377dcf2934e4dbd83fef44c314f8924"
format_stamp: "Formatted at 2026-09-27 13:49:45 on dist-test-slave-q25r"
server_key: "c81368ee8e08f0f7ed98493392d8a515"
server_key_iv: "fe75e87f805258f0b0c79c6398974a88"
server_key_version: "encryptionkey@0"
I20260927 13:49:45.603696 25452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "e377dcf2934e4dbd83fef44c314f8924"
format_stamp: "Formatted at 2026-09-27 13:49:45 on dist-test-slave-q25r"
server_key: "c81368ee8e08f0f7ed98493392d8a515"
server_key_iv: "fe75e87f805258f0b0c79c6398974a88"
server_key_version: "encryptionkey@0"
I20260927 13:49:45.609033 25452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 13:49:45.613247 25466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:45.614821 25452 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:49:45.615002 25452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "e377dcf2934e4dbd83fef44c314f8924"
format_stamp: "Formatted at 2026-09-27 13:49:45 on dist-test-slave-q25r"
server_key: "c81368ee8e08f0f7ed98493392d8a515"
server_key_iv: "fe75e87f805258f0b0c79c6398974a88"
server_key_version: "encryptionkey@0"
I20260927 13:49:45.615229 25452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:45.629140 25452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:45.633466 25452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:45.633737 25452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:45.634722 25452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:45.636098 25452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:45.636204 25452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:45.636312 25452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:45.636404 25452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 13:49:45.668133 25452 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:34935
I20260927 13:49:45.668177 25578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:34935 every 8 connection(s)
I20260927 13:49:45.669035 25452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:49:45.675025 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 25452
I20260927 13:49:45.675244 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:49:45.675793 21013 external_mini_cluster.cc:1428] Setting key e23942c4a422daddc7b26319b8f28f3f
Sep 27 13:49:45 dist-test-slave-q25r krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516985, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:45.690609 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:45.671330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:37139 (local address 127.20.133.126:32969)
0927 13:49:45.671494 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:45.671499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:45.672726 (+ 1227us) server_negotiation.cc:407] Connection header received
0927 13:49:45.674722 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:45.674727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:45.674843 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:45.674939 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:45.676984 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.677710 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.679332 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.679794 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.679908 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:45.683871 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:45.683877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:45.683880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:45.683916 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:45.686747 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:45.687682 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:45.687687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:45.687689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:45.687850 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:45.688501 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:45.688506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:45.688509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:45.688942 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:45.689091 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:45.690065 (+ 974us) server_negotiation.cc:299] Negotiation successful
0927 13:49:45.690267 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:49:45.691695 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:45.671755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:32969 (local address 127.20.133.67:37139)
0927 13:49:45.672552 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:45.672591 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:45.674392 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:45.675264 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:45.675278 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:45.676046 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:45.676782 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:45.676802 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.677887 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.677891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:45.679131 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:45.679145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.680029 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.680036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:45.680258 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:45.680982 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:45.681011 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:45.683584 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:45.686931 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:45.686940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:45.686955 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:45.687495 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:45.688023 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:45.688027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:45.688030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:45.688244 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:45.689106 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:45.689113 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:45.689599 (+ 486us) client_negotiation.cc:769] Sending connection context
0927 13:49:45.689895 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 13:49:45.690467 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":308,"threads_started":1}
I20260927 13:49:45.693063 25579 heartbeater.cc:347] Connected to a master server at 127.20.133.126:32969
I20260927 13:49:45.693394 25579 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:45.694065 25579 heartbeater.cc:511] Master 127.20.133.126:32969 requested a full tablet report, sending...
I20260927 13:49:45.695577 25126 ts_manager.cc:194] Registered new tserver with Master: e377dcf2934e4dbd83fef44c314f8924 (127.20.133.67:34935)
I20260927 13:49:45.696475 25126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:37139
I20260927 13:49:45.704211 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:49:45.712152 25314 heartbeater.cc:503] Master 127.20.133.126:32969 was elected leader, sending a full tablet report...
I20260927 13:49:45.721540 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:45.709127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.20.133.126:32969)
0927 13:49:45.709375 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:45.709382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:45.709764 (+ 382us) server_negotiation.cc:407] Connection header received
0927 13:49:45.709934 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:45.709939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:45.710012 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:45.710101 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:45.711497 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.712235 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.713675 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.714172 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.714267 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:45.716413 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:45.716418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:45.716421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:45.716452 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:45.718902 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:45.719580 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:45.719585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:45.719589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:45.719723 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:45.720076 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:45.720081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:45.720083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:45.720491 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:45.720733 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:45.721020 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 13:49:45.721229 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 13:49:45.725986 25126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:49:45.733709 25126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:49:45.772073 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:45.747385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.20.133.65:43949)
0927 13:49:45.761924 (+ 14539us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:45.761935 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:45.764016 (+ 2081us) server_negotiation.cc:407] Connection header received
0927 13:49:45.764127 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:45.764134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:45.764392 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:45.764589 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:45.765736 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.767138 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.769247 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.770563 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.770723 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:45.770925 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:45.771543 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 13:49:45.771729 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14298,"thread_start_us":14815,"threads_started":1}
I20260927 13:49:45.772106 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:45.746984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:43949 (local address 127.0.0.1:45082)
0927 13:49:45.763707 (+ 16723us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:45.763734 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:45.763902 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:45.764856 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:45.764865 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:45.764892 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:45.765498 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:45.765512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.767279 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.767285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:45.769073 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:45.769086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.771046 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.771053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:45.771263 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:45.771331 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 13:49:45.771475 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 13:49:45.771702 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16570,"mutex_wait_us":17,"thread_start_us":10580,"threads_started":1}
I20260927 13:49:45.773619 25591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:45.749319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:34935 (local address 127.0.0.1:45934)
0927 13:49:45.764041 (+ 14722us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:45.764058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:45.764192 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:45.764807 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:45.764815 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:45.764838 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:45.765493 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:45.765508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.767371 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.767376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:45.770076 (+ 2700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:45.770091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.772709 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.772717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:45.772895 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:45.772930 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:49:45.773046 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 13:49:45.773192 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14596,"mutex_wait_us":49,"spinlock_wait_cycles":8576,"thread_start_us":14072,"threads_started":1}
I20260927 13:49:45.773816 25593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:45.746858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:33531 (local address 127.0.0.1:41186)
0927 13:49:45.758031 (+ 11173us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:45.758080 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:45.758331 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:45.764542 (+ 6211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:45.764549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:45.764579 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:45.764996 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:45.765011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.766953 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.766959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:45.769551 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:45.769723 (+ 172us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.772948 (+ 3225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.772956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:45.773128 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:45.773161 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:49:45.773273 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 13:49:45.773413 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10882,"mutex_wait_us":20,"thread_start_us":16903,"threads_started":1}
I20260927 13:49:45.774487 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:45.747001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.20.133.66:33531)
0927 13:49:45.763802 (+ 16801us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:45.763811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:45.763845 (+ 34us) server_negotiation.cc:407] Connection header received
0927 13:49:45.763965 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:45.763974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:45.764281 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:45.764512 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:45.765233 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.766811 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.770160 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.772714 (+ 2554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.772867 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:45.773139 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:45.774002 (+ 863us) server_negotiation.cc:299] Negotiation successful
0927 13:49:45.774202 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16618,"thread_start_us":17011,"threads_started":1}
I20260927 13:49:45.774514 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:45.749989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.20.133.67:34935)
0927 13:49:45.763167 (+ 13178us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:45.763178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:45.764257 (+ 1079us) server_negotiation.cc:407] Connection header received
0927 13:49:45.764391 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:45.764399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:45.764595 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:45.764766 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:45.765738 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.767198 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.770285 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:45.771889 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:45.772027 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:45.772250 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:45.773988 (+ 1738us) server_negotiation.cc:299] Negotiation successful
0927 13:49:45.774194 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12951,"thread_start_us":12984,"threads_started":1}
I20260927 13:49:45.776583 25249 tablet_service.cc:1467] Processing CreateTablet for tablet 1e4a00c08da441d5a75feec7f4b67ef5 (DEFAULT_TABLE table=test-table [id=78a9fe7872fc49f8a4d8fa12dd021cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:45.778254 25249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e4a00c08da441d5a75feec7f4b67ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:45.778465 25514 tablet_service.cc:1467] Processing CreateTablet for tablet 1e4a00c08da441d5a75feec7f4b67ef5 (DEFAULT_TABLE table=test-table [id=78a9fe7872fc49f8a4d8fa12dd021cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:45.778457 25382 tablet_service.cc:1467] Processing CreateTablet for tablet 1e4a00c08da441d5a75feec7f4b67ef5 (DEFAULT_TABLE table=test-table [id=78a9fe7872fc49f8a4d8fa12dd021cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:45.779606 25514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e4a00c08da441d5a75feec7f4b67ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:45.779868 25382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e4a00c08da441d5a75feec7f4b67ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:45.792428 25597 tablet_bootstrap.cc:492] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: Bootstrap starting.
I20260927 13:49:45.795673 25597 tablet_bootstrap.cc:654] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:45.795783 25598 tablet_bootstrap.cc:492] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: Bootstrap starting.
I20260927 13:49:45.797210 25597 log.cc:824] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:45.799137 25598 tablet_bootstrap.cc:654] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:45.800444 25598 log.cc:824] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:45.801009 25597 tablet_bootstrap.cc:492] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: No bootstrap required, opened a new log
I20260927 13:49:45.801378 25597 ts_tablet_manager.cc:1397] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 13:49:45.803578 25598 tablet_bootstrap.cc:492] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: No bootstrap required, opened a new log
I20260927 13:49:45.803871 25598 ts_tablet_manager.cc:1397] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260927 13:49:45.803936 25599 tablet_bootstrap.cc:492] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: Bootstrap starting.
I20260927 13:49:45.807261 25599 tablet_bootstrap.cc:654] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:45.808431 25599 log.cc:824] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:45.811852 25599 tablet_bootstrap.cc:492] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: No bootstrap required, opened a new log
I20260927 13:49:45.812157 25599 ts_tablet_manager.cc:1397] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 13:49:45.811957 25597 raft_consensus.cc:348] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2941c414a60b45cfba7b5df6ce7783ce" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 33531 } } peers { permanent_uuid: "e4eb0593068e42b58f676755d96232f4" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43949 } } peers { permanent_uuid: "e377dcf2934e4dbd83fef44c314f8924" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 34935 } }
I20260927 13:49:45.812475 25597 raft_consensus.cc:374] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:45.812575 25597 raft_consensus.cc:729] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e377dcf2934e4dbd83fef44c314f8924, State: Initialized, Role: FOLLOWER
I20260927 13:49:45.813238 25597 consensus_queue.cc:260] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2941c414a60b45cfba7b5df6ce7783ce" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 33531 } } peers { permanent_uuid: "e4eb0593068e42b58f676755d96232f4" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43949 } } peers { permanent_uuid: "e377dcf2934e4dbd83fef44c314f8924" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 34935 } }
I20260927 13:49:45.814435 25579 heartbeater.cc:503] Master 127.20.133.126:32969 was elected leader, sending a full tablet report...
I20260927 13:49:45.814172 25598 raft_consensus.cc:348] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2941c414a60b45cfba7b5df6ce7783ce" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 33531 } } peers { permanent_uuid: "e4eb0593068e42b58f676755d96232f4" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43949 } } peers { permanent_uuid: "e377dcf2934e4dbd83fef44c314f8924" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 34935 } }
I20260927 13:49:45.814591 25598 raft_consensus.cc:374] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:45.814720 25598 raft_consensus.cc:729] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4eb0593068e42b58f676755d96232f4, State: Initialized, Role: FOLLOWER
I20260927 13:49:45.815104 25597 ts_tablet_manager.cc:1428] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260927 13:49:45.815320 25598 consensus_queue.cc:260] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2941c414a60b45cfba7b5df6ce7783ce" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 33531 } } peers { permanent_uuid: "e4eb0593068e42b58f676755d96232f4" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43949 } } peers { permanent_uuid: "e377dcf2934e4dbd83fef44c314f8924" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 34935 } }
I20260927 13:49:45.816983 25598 ts_tablet_manager.cc:1428] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 13:49:45.823983 25599 raft_consensus.cc:348] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2941c414a60b45cfba7b5df6ce7783ce" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 33531 } } peers { permanent_uuid: "e4eb0593068e42b58f676755d96232f4" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43949 } } peers { permanent_uuid: "e377dcf2934e4dbd83fef44c314f8924" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 34935 } }
I20260927 13:49:45.824594 25599 raft_consensus.cc:374] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:45.824698 25599 raft_consensus.cc:729] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2941c414a60b45cfba7b5df6ce7783ce, State: Initialized, Role: FOLLOWER
I20260927 13:49:45.825246 25599 consensus_queue.cc:260] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2941c414a60b45cfba7b5df6ce7783ce" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 33531 } } peers { permanent_uuid: "e4eb0593068e42b58f676755d96232f4" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43949 } } peers { permanent_uuid: "e377dcf2934e4dbd83fef44c314f8924" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 34935 } }
I20260927 13:49:45.826278 25447 heartbeater.cc:503] Master 127.20.133.126:32969 was elected leader, sending a full tablet report...
I20260927 13:49:45.827035 25599 ts_tablet_manager.cc:1428] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260927 13:49:45.918850 25448 tablet.cc:2369] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:49:45.920325 25580 tablet.cc:2369] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:49:45.933236 25315 tablet.cc:2369] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:49:46.021062 25605 raft_consensus.cc:482] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:49:46.021392 25605 raft_consensus.cc:504] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2941c414a60b45cfba7b5df6ce7783ce" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 33531 } } peers { permanent_uuid: "e4eb0593068e42b58f676755d96232f4" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43949 } } peers { permanent_uuid: "e377dcf2934e4dbd83fef44c314f8924" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 34935 } }
I20260927 13:49:46.023114 25605 leader_election.cc:288] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4eb0593068e42b58f676755d96232f4 (127.20.133.65:43949), e377dcf2934e4dbd83fef44c314f8924 (127.20.133.67:34935)
I20260927 13:49:46.030957 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:46.023558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:60195 (local address 127.20.133.65:43949)
0927 13:49:46.023782 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:46.023789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:46.024643 (+ 854us) server_negotiation.cc:407] Connection header received
0927 13:49:46.024757 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:46.024763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:46.024831 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:46.024900 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:46.025574 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.026356 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.029106 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.029908 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.030065 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:46.030161 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:46.030534 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 13:49:46.030683 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:49:46.031006 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:46.023758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:47039 (local address 127.20.133.67:34935)
0927 13:49:46.023973 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:46.023980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:46.024476 (+ 496us) server_negotiation.cc:407] Connection header received
0927 13:49:46.024806 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:46.024811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:46.024873 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:46.024960 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:49:46.025613 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.026714 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.029190 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.030067 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.030243 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:46.030320 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:46.030637 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 13:49:46.030780 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:49:46.031150 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:46.023533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:43949 (local address 127.20.133.66:60195)
0927 13:49:46.024513 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:46.024527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:46.024638 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:46.024940 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:46.024945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:46.024969 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:46.025411 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:46.025424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.026607 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.026618 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:46.028916 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:46.028936 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.030143 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.030150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:46.030359 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:46.030393 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:49:46.030474 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 13:49:46.030593 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":63,"thread_start_us":632,"threads_started":1}
I20260927 13:49:46.031418 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:46.023450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:34935 (local address 127.20.133.66:47039)
0927 13:49:46.024325 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:46.024378 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:46.024546 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:46.025001 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:46.025006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:46.025036 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:49:46.025470 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:46.025480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.026977 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.026982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:46.028986 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:46.029000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.030315 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.030320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:46.030468 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:46.030501 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:49:46.030595 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 13:49:46.030717 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":23,"thread_start_us":504,"threads_started":1}
I20260927 13:49:46.032265 25269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4a00c08da441d5a75feec7f4b67ef5" candidate_uuid: "2941c414a60b45cfba7b5df6ce7783ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4eb0593068e42b58f676755d96232f4" is_pre_election: true
I20260927 13:49:46.032392 25534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4a00c08da441d5a75feec7f4b67ef5" candidate_uuid: "2941c414a60b45cfba7b5df6ce7783ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e377dcf2934e4dbd83fef44c314f8924" is_pre_election: true
I20260927 13:49:46.032814 25269 raft_consensus.cc:2454] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2941c414a60b45cfba7b5df6ce7783ce in term 0.
I20260927 13:49:46.032873 25534 raft_consensus.cc:2454] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2941c414a60b45cfba7b5df6ce7783ce in term 0.
I20260927 13:49:46.033491 25336 leader_election.cc:302] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2941c414a60b45cfba7b5df6ce7783ce, e377dcf2934e4dbd83fef44c314f8924; no voters:
I20260927 13:49:46.033875 25605 raft_consensus.cc:2781] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:49:46.034016 25605 raft_consensus.cc:482] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:49:46.034107 25605 raft_consensus.cc:3037] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:46.036289 25605 raft_consensus.cc:504] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2941c414a60b45cfba7b5df6ce7783ce" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 33531 } } peers { permanent_uuid: "e4eb0593068e42b58f676755d96232f4" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43949 } } peers { permanent_uuid: "e377dcf2934e4dbd83fef44c314f8924" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 34935 } }
I20260927 13:49:46.037078 25605 leader_election.cc:288] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [CANDIDATE]: Term 1 election: Requested vote from peers e4eb0593068e42b58f676755d96232f4 (127.20.133.65:43949), e377dcf2934e4dbd83fef44c314f8924 (127.20.133.67:34935)
I20260927 13:49:46.037586 25269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4a00c08da441d5a75feec7f4b67ef5" candidate_uuid: "2941c414a60b45cfba7b5df6ce7783ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4eb0593068e42b58f676755d96232f4"
I20260927 13:49:46.037793 25534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4a00c08da441d5a75feec7f4b67ef5" candidate_uuid: "2941c414a60b45cfba7b5df6ce7783ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e377dcf2934e4dbd83fef44c314f8924"
I20260927 13:49:46.037838 25269 raft_consensus.cc:3037] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:46.038017 25534 raft_consensus.cc:3037] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:46.041323 25534 raft_consensus.cc:2454] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2941c414a60b45cfba7b5df6ce7783ce in term 1.
I20260927 13:49:46.041323 25269 raft_consensus.cc:2454] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2941c414a60b45cfba7b5df6ce7783ce in term 1.
I20260927 13:49:46.041929 25337 leader_election.cc:302] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2941c414a60b45cfba7b5df6ce7783ce, e4eb0593068e42b58f676755d96232f4; no voters:
I20260927 13:49:46.042325 25605 raft_consensus.cc:2781] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:46.042835 25605 raft_consensus.cc:686] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 1 LEADER]: Becoming Leader. State: Replica: 2941c414a60b45cfba7b5df6ce7783ce, State: Running, Role: LEADER
I20260927 13:49:46.043429 25605 consensus_queue.cc:237] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2941c414a60b45cfba7b5df6ce7783ce" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 33531 } } peers { permanent_uuid: "e4eb0593068e42b58f676755d96232f4" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43949 } } peers { permanent_uuid: "e377dcf2934e4dbd83fef44c314f8924" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 34935 } }
I20260927 13:49:46.049314 25126 catalog_manager.cc:5495] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 2941c414a60b45cfba7b5df6ce7783ce (127.20.133.66). New cstate: current_term: 1 leader_uuid: "2941c414a60b45cfba7b5df6ce7783ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2941c414a60b45cfba7b5df6ce7783ce" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 33531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4eb0593068e42b58f676755d96232f4" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e377dcf2934e4dbd83fef44c314f8924" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 34935 } health_report { overall_health: UNKNOWN } } }
I20260927 13:49:46.068619 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:46.063417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.20.133.66:33531)
0927 13:49:46.063586 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:46.063591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:46.063609 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:49:46.063780 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:46.063784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:46.063862 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:46.063990 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:49:46.064353 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.065095 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.066464 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.066898 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.066995 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:46.067501 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:49:46.068054 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:49:46.068168 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:46.068277 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 13:49:46.068388 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:49:46.077034 25269 raft_consensus.cc:1261] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4 [term 1 FOLLOWER]: Refusing update from remote peer 2941c414a60b45cfba7b5df6ce7783ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:46.077035 25534 raft_consensus.cc:1261] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924 [term 1 FOLLOWER]: Refusing update from remote peer 2941c414a60b45cfba7b5df6ce7783ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:46.078075 25608 consensus_queue.cc:1035] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4eb0593068e42b58f676755d96232f4" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:46.078409 25605 consensus_queue.cc:1035] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "e377dcf2934e4dbd83fef44c314f8924" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 34935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:46.102303 25126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:46.102567 25126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:46.106170 25126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 119a28e1d84c4ceca198c6d0e354f6bb: Sending DeleteTablet for 3 replicas of tablet 1e4a00c08da441d5a75feec7f4b67ef5
I20260927 13:49:46.107479 25382 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e4a00c08da441d5a75feec7f4b67ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:46 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:41186
I20260927 13:49:46.107487 25249 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e4a00c08da441d5a75feec7f4b67ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:46 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:45082
I20260927 13:49:46.107546 25514 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e4a00c08da441d5a75feec7f4b67ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:46 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:45934
I20260927 13:49:46.109957 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 25187
I20260927 13:49:46.120628 25622 tablet_replica.cc:330] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: stopping tablet replica
I20260927 13:49:46.121399 25622 raft_consensus.cc:2229] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:46.122124 25622 raft_consensus.cc:2258] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:46.123008 25623 tablet_replica.cc:330] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: stopping tablet replica
I20260927 13:49:46.123370 25623 raft_consensus.cc:2229] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:46.123772 25623 raft_consensus.cc:2258] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:46.126561 25622 ts_tablet_manager.cc:1905] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:46.129715 25623 ts_tablet_manager.cc:1905] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:46.132781 25622 ts_tablet_manager.cc:1918] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:49:46.133148 25622 log.cc:1196] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/1e4a00c08da441d5a75feec7f4b67ef5
I20260927 13:49:46.133704 25622 ts_tablet_manager.cc:1939] T 1e4a00c08da441d5a75feec7f4b67ef5 P e377dcf2934e4dbd83fef44c314f8924: Deleting consensus metadata
I20260927 13:49:46.135696 25111 catalog_manager.cc:4841] TS e377dcf2934e4dbd83fef44c314f8924 (127.20.133.67:34935): tablet 1e4a00c08da441d5a75feec7f4b67ef5 (table test-table [id=78a9fe7872fc49f8a4d8fa12dd021cf0]) successfully deleted
I20260927 13:49:46.136595 25623 ts_tablet_manager.cc:1918] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:49:46.136845 25623 log.cc:1196] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/1e4a00c08da441d5a75feec7f4b67ef5
I20260927 13:49:46.136932 25621 tablet_replica.cc:330] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: stopping tablet replica
I20260927 13:49:46.137459 25623 ts_tablet_manager.cc:1939] T 1e4a00c08da441d5a75feec7f4b67ef5 P e4eb0593068e42b58f676755d96232f4: Deleting consensus metadata
I20260927 13:49:46.137449 25621 raft_consensus.cc:2229] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:49:46.138168 25621 raft_consensus.cc:2258] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:46.139461 25112 catalog_manager.cc:4841] TS e4eb0593068e42b58f676755d96232f4 (127.20.133.65:43949): tablet 1e4a00c08da441d5a75feec7f4b67ef5 (table test-table [id=78a9fe7872fc49f8a4d8fa12dd021cf0]) successfully deleted
I20260927 13:49:46.140147 25621 ts_tablet_manager.cc:1905] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 13:49:46 dist-test-slave-q25r krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:46.145695 25621 ts_tablet_manager.cc:1918] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:49:46.145911 25621 log.cc:1196] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/1e4a00c08da441d5a75feec7f4b67ef5
I20260927 13:49:46.146466 25621 ts_tablet_manager.cc:1939] T 1e4a00c08da441d5a75feec7f4b67ef5 P 2941c414a60b45cfba7b5df6ce7783ce: Deleting consensus metadata
I20260927 13:49:46.148069 25110 catalog_manager.cc:4841] TS 2941c414a60b45cfba7b5df6ce7783ce (127.20.133.66:33531): tablet 1e4a00c08da441d5a75feec7f4b67ef5 (table test-table [id=78a9fe7872fc49f8a4d8fa12dd021cf0]) successfully deleted
I20260927 13:49:46.154045 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:46.111163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.20.133.65:43949)
0927 13:49:46.111354 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:46.111361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:46.134950 (+ 23589us) server_negotiation.cc:407] Connection header received
0927 13:49:46.135465 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:46.135471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:46.135565 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:46.135655 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:46.137547 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.138715 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.140666 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.141294 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.141397 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:46.146908 (+ 5511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:46.146929 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:46.146950 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:46.147009 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:46.150847 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:46.151692 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:46.151702 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:46.151707 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:46.151905 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:46.152246 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:46.152252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:46.152256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:46.152796 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:46.153116 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:46.153416 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 13:49:46.153650 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:49:46.154841 25309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:46.247274 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 25187
I20260927 13:49:46.262380 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 25320
Sep 27 13:49:46 dist-test-slave-q25r krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:49:46.278546 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:46.263189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.20.133.66:33531)
0927 13:49:46.263406 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:46.263411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:46.263432 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:49:46.263599 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:46.263604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:46.263679 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:46.263771 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:46.265040 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.265874 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.267099 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.267562 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.267675 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:46.272146 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:46.272162 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:46.272178 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:46.272221 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:46.275556 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:46.276459 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:46.276469 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:46.276473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:46.276637 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:46.277004 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:46.277007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:46.277010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:46.277392 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:46.277716 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:46.277978 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 13:49:46.278215 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 13:49:46.279296 25442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:46.392262 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 25320
I20260927 13:49:46.406474 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 25452
Sep 27 13:49:46 dist-test-slave-q25r krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:49:46.422811 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:46.407216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.20.133.67:34935)
0927 13:49:46.407443 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:46.407449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:46.407480 (+ 31us) server_negotiation.cc:407] Connection header received
0927 13:49:46.407703 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:46.407708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:46.407780 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:46.407891 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:46.409347 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.410093 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.411464 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:46.411902 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:46.412031 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:46.416747 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:46.416765 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:46.416782 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:46.416831 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:46.419955 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:46.420746 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:46.420754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:46.420757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:46.420914 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:46.421286 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:46.421290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:46.421292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:46.421649 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:46.421945 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:46.422267 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 13:49:46.422486 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 13:49:46.423457 25574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:46.513857 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 25452
I20260927 13:49:46.526641 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 25095
I20260927 13:49:46.527688 25156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:46.637019 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 25095
2026-09-27T13:49:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4291 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T13:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:46Z Disabled control of system clock
I20260927 13:49:46.711931 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:37585
--webserver_interface=127.20.133.126
--webserver_port=0
--builtin_ntp_servers=127.20.133.84:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:37585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 13:49:47.105310 25632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:47.105788 25632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:47.105898 25632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:47.114051 25632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:47.114219 25632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:47.114259 25632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:47.114315 25632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:47.114362 25632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:47.124977 25632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:37585
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:37585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:47.126762 25632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:47.128925 25632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:47.142674 25637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:47.142675 25638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:47.142568 25640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:47.144002 25632 server_base.cc:1034] running on GCE node
I20260927 13:49:47.145061 25632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:47.146895 25632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:47.148135 25632 hybrid_clock.cc:648] HybridClock initialized: now 1790516987148061 us; error 72 us; skew 500 ppm
I20260927 13:49:47.148626 25632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:49:47.151638 25632 webserver.cc:460] Webserver started at http://127.20.133.126:36975/ using document root <none> and password file <none>
I20260927 13:49:47.152575 25632 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:47.152686 25632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:47.153069 25632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:47.156663 25632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "5cff3eadd02947a1966e6f0c44f583b0"
format_stamp: "Formatted at 2026-09-27 13:49:47 on dist-test-slave-q25r"
server_key: "18a32f020b1b18f4557604deecb807ad"
server_key_iv: "66ffb2994403871d919e3d98c2c41d16"
server_key_version: "encryptionkey@0"
I20260927 13:49:47.157579 25632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "5cff3eadd02947a1966e6f0c44f583b0"
format_stamp: "Formatted at 2026-09-27 13:49:47 on dist-test-slave-q25r"
server_key: "18a32f020b1b18f4557604deecb807ad"
server_key_iv: "66ffb2994403871d919e3d98c2c41d16"
server_key_version: "encryptionkey@0"
I20260927 13:49:47.163991 25632 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 13:49:47.168385 25646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:47.170303 25632 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:49:47.170552 25632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "5cff3eadd02947a1966e6f0c44f583b0"
format_stamp: "Formatted at 2026-09-27 13:49:47 on dist-test-slave-q25r"
server_key: "18a32f020b1b18f4557604deecb807ad"
server_key_iv: "66ffb2994403871d919e3d98c2c41d16"
server_key_version: "encryptionkey@0"
I20260927 13:49:47.170818 25632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:47.178799 25632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:47.180006 25632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:47.180395 25632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:47.202051 25632 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:37585
I20260927 13:49:47.202099 25697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:37585 every 8 connection(s)
I20260927 13:49:47.203827 25632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:47.208041 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 25632
I20260927 13:49:47.208271 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:47.208523 25698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:47.209121 21013 external_mini_cluster.cc:1428] Setting key 32890528213132de7f5c2ef4c6922d87
I20260927 13:49:47.219904 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:47.211793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.20.133.126:37585)
0927 13:49:47.213138 (+ 1345us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:47.213159 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:47.213219 (+ 60us) server_negotiation.cc:407] Connection header received
0927 13:49:47.215292 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:47.215326 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:47.215992 (+ 666us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:47.216522 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:47.217829 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:47.217867 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:47.217884 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:47.217901 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:47.217976 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:47.217997 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:47.218213 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:47.218325 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 13:49:47.218674 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":488,"threads_started":1}
I20260927 13:49:47.223677 25698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0: Bootstrap starting.
I20260927 13:49:47.227020 25698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:47.228130 25698 log.cc:824] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:47.231266 25698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0: No bootstrap required, opened a new log
I20260927 13:49:47.238924 25698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cff3eadd02947a1966e6f0c44f583b0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37585 } }
I20260927 13:49:47.239331 25698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:47.239411 25698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cff3eadd02947a1966e6f0c44f583b0, State: Initialized, Role: FOLLOWER
I20260927 13:49:47.240031 25698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cff3eadd02947a1966e6f0c44f583b0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37585 } }
I20260927 13:49:47.240262 25698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:47.240411 25698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:47.240561 25698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:47.242674 25698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cff3eadd02947a1966e6f0c44f583b0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37585 } }
I20260927 13:49:47.243180 25698 leader_election.cc:302] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cff3eadd02947a1966e6f0c44f583b0; no voters:
I20260927 13:49:47.243867 25698 leader_election.cc:288] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:47.244127 25703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:47.245283 25703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [term 1 LEADER]: Becoming Leader. State: Replica: 5cff3eadd02947a1966e6f0c44f583b0, State: Running, Role: LEADER
I20260927 13:49:47.246073 25703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cff3eadd02947a1966e6f0c44f583b0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37585 } }
I20260927 13:49:47.246291 25698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:47.249914 25704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cff3eadd02947a1966e6f0c44f583b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cff3eadd02947a1966e6f0c44f583b0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37585 } } }
I20260927 13:49:47.250450 25704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:47.249867 25705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cff3eadd02947a1966e6f0c44f583b0. Latest consensus state: current_term: 1 leader_uuid: "5cff3eadd02947a1966e6f0c44f583b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cff3eadd02947a1966e6f0c44f583b0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37585 } } }
I20260927 13:49:47.250852 25705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:47.253108 25712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:47.259685 25712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:47.270994 25712 catalog_manager.cc:1347] Generated new cluster ID: 60a47b3bf04f43ffb1c8799dfd9d2e7b
I20260927 13:49:47.271131 25712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:47.308642 25712 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:47.309360 25712 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:47.318567 25712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0: Generated new TSK 0
I20260927 13:49:47.319954 25712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:49:47.326390 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37585
--builtin_ntp_servers=127.20.133.84:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 13:49:47.668957 25722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:47.669246 25722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:47.669358 25722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:47.677657 25722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:47.677769 25722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:47.677908 25722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:49:47.687844 25722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:47.689225 25722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:47.691089 25722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:47.704027 25727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:47.703995 25730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:47.704064 25728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:47.704797 25722 server_base.cc:1034] running on GCE node
I20260927 13:49:47.705379 25722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:47.706319 25722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:47.707513 25722 hybrid_clock.cc:648] HybridClock initialized: now 1790516987707457 us; error 53 us; skew 500 ppm
I20260927 13:49:47.707820 25722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:49:47.710093 25722 webserver.cc:460] Webserver started at http://127.20.133.65:40595/ using document root <none> and password file <none>
I20260927 13:49:47.710795 25722 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:47.710891 25722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:47.711197 25722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:47.714043 25722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "179f201df68b47edace0a023b7c38077"
format_stamp: "Formatted at 2026-09-27 13:49:47 on dist-test-slave-q25r"
server_key: "2d3d668060d7bf82d52f5ee7ea8076d3"
server_key_iv: "85c67d595a39c1144cbc65a4e4ed5f4b"
server_key_version: "encryptionkey@0"
I20260927 13:49:47.714830 25722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "179f201df68b47edace0a023b7c38077"
format_stamp: "Formatted at 2026-09-27 13:49:47 on dist-test-slave-q25r"
server_key: "2d3d668060d7bf82d52f5ee7ea8076d3"
server_key_iv: "85c67d595a39c1144cbc65a4e4ed5f4b"
server_key_version: "encryptionkey@0"
I20260927 13:49:47.719734 25722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 13:49:47.723235 25736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:47.724712 25722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 13:49:47.724870 25722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "179f201df68b47edace0a023b7c38077"
format_stamp: "Formatted at 2026-09-27 13:49:47 on dist-test-slave-q25r"
server_key: "2d3d668060d7bf82d52f5ee7ea8076d3"
server_key_iv: "85c67d595a39c1144cbc65a4e4ed5f4b"
server_key_version: "encryptionkey@0"
I20260927 13:49:47.725095 25722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:47.730598 25722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:47.731381 25722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:47.731683 25722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:47.732648 25722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:47.734378 25722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:47.734499 25722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:47.734608 25722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:47.734680 25722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:47.765352 25722 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:36709
I20260927 13:49:47.765404 25848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:36709 every 8 connection(s)
I20260927 13:49:47.766598 25722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:49:47.767331 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 25722
I20260927 13:49:47.767513 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:49:47.768219 21013 external_mini_cluster.cc:1428] Setting key 07174caa4afd95a8ff0574cdc0aa5cf9
I20260927 13:49:47.771296 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37585
--builtin_ntp_servers=127.20.133.84:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 13:49:47.776072 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:47.769203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:51001 (local address 127.20.133.126:37585)
0927 13:49:47.769652 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:47.769658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:47.770919 (+ 1261us) server_negotiation.cc:407] Connection header received
0927 13:49:47.773290 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:47.773295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:47.773353 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:47.773467 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:47.775079 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:47.775084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:47.775087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:47.775107 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:47.775159 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:47.775173 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:47.775300 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:47.775715 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 13:49:47.775819 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":215,"threads_started":1}
I20260927 13:49:47.777213 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:47.769749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37585 (local address 127.20.133.65:51001)
0927 13:49:47.770757 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:47.770808 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:47.772954 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:47.773819 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:47.773831 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:47.773959 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:47.774708 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:49:47.774711 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:47.774761 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:49:47.774874 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:47.775286 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:47.775295 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:47.775304 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 13:49:47.775642 (+ 338us) client_negotiation.cc:240] Negotiation successful
0927 13:49:47.775952 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":262,"threads_started":1}
I20260927 13:49:47.778779 25849 heartbeater.cc:347] Connected to a master server at 127.20.133.126:37585
I20260927 13:49:47.779058 25849 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:47.779627 25849 heartbeater.cc:511] Master 127.20.133.126:37585 requested a full tablet report, sending...
I20260927 13:49:47.781749 25663 ts_manager.cc:194] Registered new tserver with Master: 179f201df68b47edace0a023b7c38077 (127.20.133.65:36709)
W20260927 13:49:48.085830 25853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:48.086150 25853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:48.086311 25853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:48.094235 25853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:48.094386 25853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:48.094530 25853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:49:48.104233 25853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:48.105578 25853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:48.107375 25853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:48.119817 25858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:48.119954 25859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:48.119890 25861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:48.120663 25853 server_base.cc:1034] running on GCE node
I20260927 13:49:48.121227 25853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:48.122104 25853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:48.123309 25853 hybrid_clock.cc:648] HybridClock initialized: now 1790516988123255 us; error 49 us; skew 500 ppm
I20260927 13:49:48.123656 25853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:49:48.125869 25853 webserver.cc:460] Webserver started at http://127.20.133.66:45011/ using document root <none> and password file <none>
I20260927 13:49:48.126570 25853 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:48.126662 25853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:48.126978 25853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:48.130084 25853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "fc47ab531b224f878efc4217765770b3"
format_stamp: "Formatted at 2026-09-27 13:49:48 on dist-test-slave-q25r"
server_key: "bfc365865898e12c46ce0ccd35962803"
server_key_iv: "7d342648fb4dd05dd8ca37ab95a9ba71"
server_key_version: "encryptionkey@0"
I20260927 13:49:48.130888 25853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "fc47ab531b224f878efc4217765770b3"
format_stamp: "Formatted at 2026-09-27 13:49:48 on dist-test-slave-q25r"
server_key: "bfc365865898e12c46ce0ccd35962803"
server_key_iv: "7d342648fb4dd05dd8ca37ab95a9ba71"
server_key_version: "encryptionkey@0"
I20260927 13:49:48.136039 25853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:49:48.140137 25867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:48.141778 25853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:49:48.141976 25853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "fc47ab531b224f878efc4217765770b3"
format_stamp: "Formatted at 2026-09-27 13:49:48 on dist-test-slave-q25r"
server_key: "bfc365865898e12c46ce0ccd35962803"
server_key_iv: "7d342648fb4dd05dd8ca37ab95a9ba71"
server_key_version: "encryptionkey@0"
I20260927 13:49:48.142220 25853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:48.148182 25853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:48.149080 25853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:48.149341 25853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:48.150336 25853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:48.151835 25853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:48.151933 25853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:48.152040 25853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:48.152108 25853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:48.182121 25853 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:37607
I20260927 13:49:48.182152 25979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:37607 every 8 connection(s)
I20260927 13:49:48.183503 25853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:49:48.191766 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.186003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:42925 (local address 127.20.133.126:37585)
0927 13:49:48.186198 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:48.186203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:48.187625 (+ 1422us) server_negotiation.cc:407] Connection header received
0927 13:49:48.189443 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:48.189449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:48.189525 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:48.189634 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:48.191053 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:48.191058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:48.191061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:48.191077 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:48.191110 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:48.191122 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:48.191232 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:48.191414 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 13:49:48.191481 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:49:48.192786 25982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.186518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37585 (local address 127.20.133.66:42925)
0927 13:49:48.187449 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:48.187487 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:48.189166 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:48.189882 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:48.189891 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:48.189985 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:48.190725 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:49:48.190728 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:48.190766 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:49:48.190855 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:48.191213 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:48.191224 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:48.191232 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:49:48.191488 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 13:49:48.191764 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":279,"threads_started":1}
I20260927 13:49:48.193892 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 25853
I20260927 13:49:48.194049 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:49:48.194331 25980 heartbeater.cc:347] Connected to a master server at 127.20.133.126:37585
I20260927 13:49:48.194511 21013 external_mini_cluster.cc:1428] Setting key 95e94fac72b2cb066ce426e71fbc0229
I20260927 13:49:48.194624 25980 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:48.195279 25980 heartbeater.cc:511] Master 127.20.133.126:37585 requested a full tablet report, sending...
I20260927 13:49:48.196655 25663 ts_manager.cc:194] Registered new tserver with Master: fc47ab531b224f878efc4217765770b3 (127.20.133.66:37607)
I20260927 13:49:48.197288 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37585
--builtin_ntp_servers=127.20.133.84:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 13:49:48.521879 25983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:48.522172 25983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:48.522274 25983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:48.529745 25983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:48.529861 25983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:48.529973 25983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:49:48.539633 25983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:48.540962 25983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:48.542690 25983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:48.555708 25991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:48.555805 25988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:48.555797 25989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:48.556962 25983 server_base.cc:1034] running on GCE node
I20260927 13:49:48.557574 25983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:48.558539 25983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:48.559772 25983 hybrid_clock.cc:648] HybridClock initialized: now 1790516988559721 us; error 41 us; skew 500 ppm
I20260927 13:49:48.560114 25983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:49:48.562470 25983 webserver.cc:460] Webserver started at http://127.20.133.67:36403/ using document root <none> and password file <none>
I20260927 13:49:48.563174 25983 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:48.563264 25983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:48.563601 25983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:48.566471 25983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "edbf4f92449e49da9ad79e2326a0b683"
format_stamp: "Formatted at 2026-09-27 13:49:48 on dist-test-slave-q25r"
server_key: "ae86ccb29c6d35ba2befebc703e63236"
server_key_iv: "c15ea54d9fb24ef5b87972ea459a933a"
server_key_version: "encryptionkey@0"
I20260927 13:49:48.567277 25983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "edbf4f92449e49da9ad79e2326a0b683"
format_stamp: "Formatted at 2026-09-27 13:49:48 on dist-test-slave-q25r"
server_key: "ae86ccb29c6d35ba2befebc703e63236"
server_key_iv: "c15ea54d9fb24ef5b87972ea459a933a"
server_key_version: "encryptionkey@0"
I20260927 13:49:48.572427 25983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 13:49:48.575995 25997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:48.577487 25983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:49:48.577642 25983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "edbf4f92449e49da9ad79e2326a0b683"
format_stamp: "Formatted at 2026-09-27 13:49:48 on dist-test-slave-q25r"
server_key: "ae86ccb29c6d35ba2befebc703e63236"
server_key_iv: "c15ea54d9fb24ef5b87972ea459a933a"
server_key_version: "encryptionkey@0"
I20260927 13:49:48.577857 25983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:48.583403 25983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:48.584233 25983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:48.584507 25983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:48.585412 25983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:48.586807 25983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:48.586903 25983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:48.587008 25983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:48.587082 25983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:48.617125 25983 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:35731
I20260927 13:49:48.617161 26109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:35731 every 8 connection(s)
I20260927 13:49:48.618460 25983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:49:48.626224 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.620858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:56763 (local address 127.20.133.126:37585)
0927 13:49:48.621095 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:48.621100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:48.622425 (+ 1325us) server_negotiation.cc:407] Connection header received
0927 13:49:48.624161 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:48.624166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:48.624236 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:48.624352 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:48.625582 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:48.625586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:48.625589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:48.625603 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:48.625630 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:48.625641 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:48.625737 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:48.625945 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 13:49:48.625996 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 13:49:48.627162 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.621364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37585 (local address 127.20.133.67:56763)
0927 13:49:48.622273 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:48.622315 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:48.623873 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:48.624576 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:48.624585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:48.624683 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:48.625325 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:49:48.625327 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:48.625364 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:49:48.625451 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:48.625755 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:48.625765 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:48.625772 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 13:49:48.626007 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 13:49:48.626219 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":288,"threads_started":1}
I20260927 13:49:48.628043 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 25983
I20260927 13:49:48.628227 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:49:48.628705 26110 heartbeater.cc:347] Connected to a master server at 127.20.133.126:37585
I20260927 13:49:48.628763 21013 external_mini_cluster.cc:1428] Setting key 84ace698b6471f9001c5c1ed29cc181c
I20260927 13:49:48.629031 26110 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:48.629745 26110 heartbeater.cc:511] Master 127.20.133.126:37585 requested a full tablet report, sending...
I20260927 13:49:48.631243 25662 ts_manager.cc:194] Registered new tserver with Master: edbf4f92449e49da9ad79e2326a0b683 (127.20.133.67:35731)
I20260927 13:49:48.644052 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:49:48.654743 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.648776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.20.133.126:37585)
0927 13:49:48.648947 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:48.648952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:48.649440 (+ 488us) server_negotiation.cc:407] Connection header received
0927 13:49:48.649609 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:48.649614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:48.649670 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:48.649755 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:48.654081 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:48.654086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:48.654089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:48.654111 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:48.654153 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:48.654168 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:48.654312 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:48.654410 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 13:49:48.654479 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:49:48.658967 25662 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:49:48.666143 25662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:49:48.694263 26122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.678346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:37607 (local address 127.0.0.1:35014)
0927 13:49:48.690174 (+ 11828us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:48.690230 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:48.690378 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:48.691130 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:48.691137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:48.691241 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:48.692979 (+ 1738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:49:48.692983 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:48.693013 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:49:48.693095 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:48.693546 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:48.693558 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:48.693566 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:49:48.693725 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 13:49:48.693793 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11551,"thread_start_us":11386,"threads_started":1}
I20260927 13:49:48.695448 26123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.678624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.20.133.66:37607)
0927 13:49:48.689699 (+ 11075us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:48.689711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:48.690553 (+ 842us) server_negotiation.cc:407] Connection header received
0927 13:49:48.690639 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:48.690648 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:48.690935 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:48.691802 (+ 867us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:48.693281 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:48.693299 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:48.693317 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:48.693344 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:48.693412 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:48.693431 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:48.694971 (+ 1540us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:48.695045 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 13:49:48.695153 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10856,"thread_start_us":10709,"threads_started":1}
I20260927 13:49:48.696377 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.679190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52406 (local address 127.20.133.67:35731)
0927 13:49:48.691883 (+ 12693us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:48.691891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:48.691921 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:49:48.691982 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:48.691989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:48.692162 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:48.692312 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:48.694319 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:48.694329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:48.694342 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:48.694361 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:48.694418 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:48.694440 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:48.694587 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:48.695900 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0927 13:49:48.695990 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12537,"thread_start_us":12374,"threads_started":1}
I20260927 13:49:48.697016 26124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.678641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:35731 (local address 127.0.0.1:52406)
0927 13:49:48.691680 (+ 13039us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:48.691703 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:48.691802 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:48.693266 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:48.693270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:48.693292 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:48.694045 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:49:48.694048 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:48.694065 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:49:48.694111 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:48.694678 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:48.694682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:48.694687 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:49:48.696658 (+ 1971us) client_negotiation.cc:240] Negotiation successful
0927 13:49:48.696722 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12894,"thread_start_us":13222,"threads_started":1}
I20260927 13:49:48.699481 26122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.692223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36709 (local address 127.0.0.1:33384)
0927 13:49:48.694732 (+ 2509us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:48.694749 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:48.694812 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:48.698097 (+ 3285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:48.698102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:48.698136 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:48.698670 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:49:48.698673 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:48.698689 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:49:48.698732 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:48.699102 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:48.699106 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:48.699110 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 13:49:48.699208 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 13:49:48.699258 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2402,"thread_start_us":5065,"threads_started":1}
I20260927 13:49:48.699627 26127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.692596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.20.133.65:36709)
0927 13:49:48.697705 (+ 5109us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:48.697711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:48.697733 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:49:48.697784 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:48.697790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:48.697930 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:48.698043 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:48.698883 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:48.698890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:48.698901 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:48.698925 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:48.698975 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:48.698994 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:48.699129 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:48.699250 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 13:49:48.699335 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4964,"thread_start_us":6113,"threads_started":1}
I20260927 13:49:48.699254 25915 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4dc6ba7f404f7ba7bab181def8addc (DEFAULT_TABLE table=test-table [id=1771b5c4da164338a9fc61b472f77830]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:48.700646 25915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4dc6ba7f404f7ba7bab181def8addc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:48.702380 26045 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4dc6ba7f404f7ba7bab181def8addc (DEFAULT_TABLE table=test-table [id=1771b5c4da164338a9fc61b472f77830]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:48.703608 26045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4dc6ba7f404f7ba7bab181def8addc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:48.703750 25784 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4dc6ba7f404f7ba7bab181def8addc (DEFAULT_TABLE table=test-table [id=1771b5c4da164338a9fc61b472f77830]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:49:48.705034 25784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4dc6ba7f404f7ba7bab181def8addc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:48.716427 26128 tablet_bootstrap.cc:492] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: Bootstrap starting.
I20260927 13:49:48.717706 26129 tablet_bootstrap.cc:492] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: Bootstrap starting.
I20260927 13:49:48.719610 26130 tablet_bootstrap.cc:492] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077: Bootstrap starting.
I20260927 13:49:48.721287 26128 tablet_bootstrap.cc:654] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:48.721354 26129 tablet_bootstrap.cc:654] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:48.722265 26128 log.cc:824] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:48.722517 26129 log.cc:824] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:48.722924 26130 tablet_bootstrap.cc:654] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:48.724159 26130 log.cc:824] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:48.727206 26128 tablet_bootstrap.cc:492] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: No bootstrap required, opened a new log
I20260927 13:49:48.727396 26130 tablet_bootstrap.cc:492] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077: No bootstrap required, opened a new log
I20260927 13:49:48.727394 26129 tablet_bootstrap.cc:492] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: No bootstrap required, opened a new log
I20260927 13:49:48.727483 26128 ts_tablet_manager.cc:1397] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260927 13:49:48.727651 26129 ts_tablet_manager.cc:1397] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 13:49:48.727651 26130 ts_tablet_manager.cc:1397] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:49:48.735159 26128 raft_consensus.cc:348] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc47ab531b224f878efc4217765770b3" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37607 } } peers { permanent_uuid: "edbf4f92449e49da9ad79e2326a0b683" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35731 } } peers { permanent_uuid: "179f201df68b47edace0a023b7c38077" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36709 } }
I20260927 13:49:48.735579 26128 raft_consensus.cc:374] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:48.735698 26128 raft_consensus.cc:729] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edbf4f92449e49da9ad79e2326a0b683, State: Initialized, Role: FOLLOWER
I20260927 13:49:48.736305 26128 consensus_queue.cc:260] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc47ab531b224f878efc4217765770b3" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37607 } } peers { permanent_uuid: "edbf4f92449e49da9ad79e2326a0b683" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35731 } } peers { permanent_uuid: "179f201df68b47edace0a023b7c38077" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36709 } }
I20260927 13:49:48.737551 26110 heartbeater.cc:503] Master 127.20.133.126:37585 was elected leader, sending a full tablet report...
I20260927 13:49:48.738092 26128 ts_tablet_manager.cc:1428] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 13:49:48.739010 26129 raft_consensus.cc:348] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc47ab531b224f878efc4217765770b3" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37607 } } peers { permanent_uuid: "edbf4f92449e49da9ad79e2326a0b683" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35731 } } peers { permanent_uuid: "179f201df68b47edace0a023b7c38077" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36709 } }
I20260927 13:49:48.739508 26129 raft_consensus.cc:374] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:48.739604 26129 raft_consensus.cc:729] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc47ab531b224f878efc4217765770b3, State: Initialized, Role: FOLLOWER
I20260927 13:49:48.739492 26130 raft_consensus.cc:348] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc47ab531b224f878efc4217765770b3" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37607 } } peers { permanent_uuid: "edbf4f92449e49da9ad79e2326a0b683" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35731 } } peers { permanent_uuid: "179f201df68b47edace0a023b7c38077" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36709 } }
I20260927 13:49:48.739918 26130 raft_consensus.cc:374] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:48.740003 26130 raft_consensus.cc:729] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179f201df68b47edace0a023b7c38077, State: Initialized, Role: FOLLOWER
I20260927 13:49:48.740352 26129 consensus_queue.cc:260] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc47ab531b224f878efc4217765770b3" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37607 } } peers { permanent_uuid: "edbf4f92449e49da9ad79e2326a0b683" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35731 } } peers { permanent_uuid: "179f201df68b47edace0a023b7c38077" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36709 } }
I20260927 13:49:48.740586 26130 consensus_queue.cc:260] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc47ab531b224f878efc4217765770b3" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37607 } } peers { permanent_uuid: "edbf4f92449e49da9ad79e2326a0b683" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35731 } } peers { permanent_uuid: "179f201df68b47edace0a023b7c38077" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36709 } }
I20260927 13:49:48.741568 25980 heartbeater.cc:503] Master 127.20.133.126:37585 was elected leader, sending a full tablet report...
I20260927 13:49:48.742311 26129 ts_tablet_manager.cc:1428] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 13:49:48.743233 25849 heartbeater.cc:503] Master 127.20.133.126:37585 was elected leader, sending a full tablet report...
I20260927 13:49:48.744697 26130 ts_tablet_manager.cc:1428] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260927 13:49:48.768970 25850 tablet.cc:2369] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:49:48.870131 26111 tablet.cc:2369] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:49:48.935513 25981 tablet.cc:2369] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:49:48.972295 26136 raft_consensus.cc:482] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:49:48.972671 26136 raft_consensus.cc:504] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc47ab531b224f878efc4217765770b3" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37607 } } peers { permanent_uuid: "edbf4f92449e49da9ad79e2326a0b683" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35731 } } peers { permanent_uuid: "179f201df68b47edace0a023b7c38077" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36709 } }
I20260927 13:49:48.974284 26136 leader_election.cc:288] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc47ab531b224f878efc4217765770b3 (127.20.133.66:37607), edbf4f92449e49da9ad79e2326a0b683 (127.20.133.67:35731)
I20260927 13:49:48.978398 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.974775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:39343 (local address 127.20.133.67:35731)
0927 13:49:48.974992 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:48.975001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:48.976274 (+ 1273us) server_negotiation.cc:407] Connection header received
0927 13:49:48.976444 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:48.976448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:48.976497 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:48.976569 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:48.977738 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:48.977744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:48.977748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:48.977767 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:48.977804 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:48.977819 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:48.977939 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:48.978079 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 13:49:48.978139 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 13:49:48.978398 26123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.975018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:56261 (local address 127.20.133.66:37607)
0927 13:49:48.975274 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:48.975281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:48.975869 (+ 588us) server_negotiation.cc:407] Connection header received
0927 13:49:48.976173 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:48.976180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:48.976263 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:48.976370 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:48.977738 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:48.977744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:48.977748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:48.977767 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:48.977804 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:48.977819 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:48.977933 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:48.978079 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 13:49:48.978139 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260927 13:49:48.978781 26138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.974743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:35731 (local address 127.20.133.65:39343)
0927 13:49:48.976160 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:48.976183 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:48.976304 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:48.976605 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:48.976610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:48.976627 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:48.977488 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:49:48.977497 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:48.977518 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:49:48.977571 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:48.977944 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:48.977950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:48.977956 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 13:49:48.978139 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 13:49:48.978214 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1302,"thread_start_us":1081,"threads_started":1}
I20260927 13:49:48.978787 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:48.974637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:37607 (local address 127.20.133.65:56261)
0927 13:49:48.975741 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:48.975765 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:48.975993 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:48.976584 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:48.976589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:48.976614 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:48.977544 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:49:48.977547 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:48.977560 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:49:48.977591 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:48.977944 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:48.977950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:48.977959 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 13:49:48.978163 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 13:49:48.978226 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":20,"thread_start_us":722,"threads_started":1}
I20260927 13:49:48.979722 26065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4dc6ba7f404f7ba7bab181def8addc" candidate_uuid: "179f201df68b47edace0a023b7c38077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edbf4f92449e49da9ad79e2326a0b683" is_pre_election: true
I20260927 13:49:48.979717 25935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4dc6ba7f404f7ba7bab181def8addc" candidate_uuid: "179f201df68b47edace0a023b7c38077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc47ab531b224f878efc4217765770b3" is_pre_election: true
I20260927 13:49:48.980217 26065 raft_consensus.cc:2454] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 179f201df68b47edace0a023b7c38077 in term 0.
I20260927 13:49:48.980258 25935 raft_consensus.cc:2454] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 179f201df68b47edace0a023b7c38077 in term 0.
I20260927 13:49:48.980906 25737 leader_election.cc:302] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179f201df68b47edace0a023b7c38077, edbf4f92449e49da9ad79e2326a0b683; no voters:
I20260927 13:49:48.981328 26136 raft_consensus.cc:2781] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:49:48.981482 26136 raft_consensus.cc:482] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:49:48.981581 26136 raft_consensus.cc:3037] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:48.983814 26136 raft_consensus.cc:504] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc47ab531b224f878efc4217765770b3" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37607 } } peers { permanent_uuid: "edbf4f92449e49da9ad79e2326a0b683" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35731 } } peers { permanent_uuid: "179f201df68b47edace0a023b7c38077" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36709 } }
I20260927 13:49:48.984447 26136 leader_election.cc:288] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [CANDIDATE]: Term 1 election: Requested vote from peers fc47ab531b224f878efc4217765770b3 (127.20.133.66:37607), edbf4f92449e49da9ad79e2326a0b683 (127.20.133.67:35731)
I20260927 13:49:48.984884 25935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4dc6ba7f404f7ba7bab181def8addc" candidate_uuid: "179f201df68b47edace0a023b7c38077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc47ab531b224f878efc4217765770b3"
I20260927 13:49:48.984911 26065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4dc6ba7f404f7ba7bab181def8addc" candidate_uuid: "179f201df68b47edace0a023b7c38077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edbf4f92449e49da9ad79e2326a0b683"
I20260927 13:49:48.985105 26065 raft_consensus.cc:3037] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:48.985100 25935 raft_consensus.cc:3037] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:48.988240 25935 raft_consensus.cc:2454] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 179f201df68b47edace0a023b7c38077 in term 1.
I20260927 13:49:48.988240 26065 raft_consensus.cc:2454] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 179f201df68b47edace0a023b7c38077 in term 1.
I20260927 13:49:48.988847 25738 leader_election.cc:302] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179f201df68b47edace0a023b7c38077, fc47ab531b224f878efc4217765770b3; no voters:
I20260927 13:49:48.989205 26136 raft_consensus.cc:2781] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:48.989675 26136 raft_consensus.cc:686] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 1 LEADER]: Becoming Leader. State: Replica: 179f201df68b47edace0a023b7c38077, State: Running, Role: LEADER
I20260927 13:49:48.990226 26136 consensus_queue.cc:237] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc47ab531b224f878efc4217765770b3" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37607 } } peers { permanent_uuid: "edbf4f92449e49da9ad79e2326a0b683" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35731 } } peers { permanent_uuid: "179f201df68b47edace0a023b7c38077" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36709 } }
I20260927 13:49:48.994896 25662 catalog_manager.cc:5495] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 179f201df68b47edace0a023b7c38077 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "179f201df68b47edace0a023b7c38077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc47ab531b224f878efc4217765770b3" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edbf4f92449e49da9ad79e2326a0b683" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "179f201df68b47edace0a023b7c38077" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36709 } health_report { overall_health: HEALTHY } } }
I20260927 13:49:49.055441 26127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:49.053306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.20.133.65:36709)
0927 13:49:49.053491 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:49.053497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:49.053515 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:49:49.053783 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:49.053788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:49.053912 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:49.053997 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:49.054813 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:49.054818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:49.054821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:49.054856 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:49.054889 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:49.054902 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:49.055003 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:49.055112 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 13:49:49.055177 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:49:49.064167 26065 raft_consensus.cc:1261] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683 [term 1 FOLLOWER]: Refusing update from remote peer 179f201df68b47edace0a023b7c38077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:49.064167 25935 raft_consensus.cc:1261] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3 [term 1 FOLLOWER]: Refusing update from remote peer 179f201df68b47edace0a023b7c38077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:49:49.065132 26139 consensus_queue.cc:1035] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edbf4f92449e49da9ad79e2326a0b683" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:49.065455 26136 consensus_queue.cc:1035] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc47ab531b224f878efc4217765770b3" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 37607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:49:49.076468 26142 mvcc.cc:205] Tried to move back new op lower bound from 7333957587197657088 to 7333957586907897856. Current Snapshot: MvccSnapshot[applied={T|T < 7333957587197657088}]
I20260927 13:49:49.078655 26144 mvcc.cc:205] Tried to move back new op lower bound from 7333957587197657088 to 7333957586907897856. Current Snapshot: MvccSnapshot[applied={T|T < 7333957587197657088}]
I20260927 13:49:49.102921 25662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:49.103227 25662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:49:49.106828 25662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cff3eadd02947a1966e6f0c44f583b0: Sending DeleteTablet for 3 replicas of tablet 7a4dc6ba7f404f7ba7bab181def8addc
I20260927 13:49:49.108153 25784 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4dc6ba7f404f7ba7bab181def8addc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:49 UTC) from {username='slave'} at 127.0.0.1:33384
I20260927 13:49:49.108112 25915 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4dc6ba7f404f7ba7bab181def8addc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:49 UTC) from {username='slave'} at 127.0.0.1:35014
I20260927 13:49:49.108209 26045 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4dc6ba7f404f7ba7bab181def8addc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:49:49 UTC) from {username='slave'} at 127.0.0.1:52406
I20260927 13:49:49.110445 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 25722
I20260927 13:49:49.117612 26154 tablet_replica.cc:330] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: stopping tablet replica
I20260927 13:49:49.118247 26154 raft_consensus.cc:2229] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:49.118898 26154 raft_consensus.cc:2258] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:49.120429 26153 tablet_replica.cc:330] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077: stopping tablet replica
I20260927 13:49:49.120534 26127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:49.112270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.20.133.65:36709)
0927 13:49:49.112811 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:49.112818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:49.116913 (+ 4095us) server_negotiation.cc:407] Connection header received
0927 13:49:49.117172 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:49.117179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:49.117255 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:49.117367 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:49.119425 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:49.119433 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:49.119439 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:49.119465 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:49.119519 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:49.119537 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:49.119682 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:49.120129 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 13:49:49.120218 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408}
I20260927 13:49:49.121584 26154 ts_tablet_manager.cc:1905] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:49.121785 26152 tablet_replica.cc:330] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: stopping tablet replica
I20260927 13:49:49.122104 25844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:49.122287 26152 raft_consensus.cc:2229] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:49:49.122856 26152 raft_consensus.cc:2258] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:49.125865 26152 ts_tablet_manager.cc:1905] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:49:49.129359 26154 ts_tablet_manager.cc:1918] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:49:49.129534 26154 log.cc:1196] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/7a4dc6ba7f404f7ba7bab181def8addc
I20260927 13:49:49.130023 26154 ts_tablet_manager.cc:1939] T 7a4dc6ba7f404f7ba7bab181def8addc P edbf4f92449e49da9ad79e2326a0b683: Deleting consensus metadata
I20260927 13:49:49.131428 26152 ts_tablet_manager.cc:1918] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:49:49.131572 26152 log.cc:1196] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/7a4dc6ba7f404f7ba7bab181def8addc
I20260927 13:49:49.131932 25647 catalog_manager.cc:4841] TS edbf4f92449e49da9ad79e2326a0b683 (127.20.133.67:35731): tablet 7a4dc6ba7f404f7ba7bab181def8addc (table test-table [id=1771b5c4da164338a9fc61b472f77830]) successfully deleted
I20260927 13:49:49.132088 26152 ts_tablet_manager.cc:1939] T 7a4dc6ba7f404f7ba7bab181def8addc P fc47ab531b224f878efc4217765770b3: Deleting consensus metadata
I20260927 13:49:49.133466 25648 catalog_manager.cc:4841] TS fc47ab531b224f878efc4217765770b3 (127.20.133.66:37607): tablet 7a4dc6ba7f404f7ba7bab181def8addc (table test-table [id=1771b5c4da164338a9fc61b472f77830]) successfully deleted
I20260927 13:49:49.122673 26153 raft_consensus.cc:2229] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:49:49.225533 26153 raft_consensus.cc:2258] T 7a4dc6ba7f404f7ba7bab181def8addc P 179f201df68b47edace0a023b7c38077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:49:49.226058 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 25722
I20260927 13:49:49.239791 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 25853
W20260927 13:49:49.239765 25647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.133.65:36709 (error 108)
I20260927 13:49:49.241619 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:49.240960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36709 (local address 127.0.0.1:33412)
0927 13:49:49.241363 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:49.241466 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":197,"threads_started":1}
W20260927 13:49:49.241829 25647 catalog_manager.cc:4584] TS 179f201df68b47edace0a023b7c38077 (127.20.133.65:36709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a4dc6ba7f404f7ba7bab181def8addc: Network error: Client connection negotiation failed: client connection to 127.20.133.65:36709: connect: Connection refused (error 111)
I20260927 13:49:49.243106 26123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:49.240539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.20.133.66:37607)
0927 13:49:49.240752 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:49.240758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:49.240825 (+ 67us) server_negotiation.cc:407] Connection header received
0927 13:49:49.240987 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:49.240991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:49.241080 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:49.241193 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:49.242338 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:49.242342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:49.242346 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:49.242361 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:49.242392 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:49.242403 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:49.242506 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:49.242706 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 13:49:49.242771 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:49:49.243772 25975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:49.295562 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:49.295119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36709 (local address 127.0.0.1:33418)
0927 13:49:49.295328 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:49.295422 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 13:49:49.296629 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:49.296289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36709 (local address 127.0.0.1:33424)
0927 13:49:49.296425 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:49.296498 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 13:49:49.332082 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 25853
I20260927 13:49:49.345425 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 25983
I20260927 13:49:49.348270 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:49.346222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.20.133.67:35731)
0927 13:49:49.346384 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:49.346394 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:49.346415 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:49:49.346473 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:49.346476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:49.346572 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:49.346675 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:49.347645 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:49.347650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:49.347653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:49.347670 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:49.347710 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:49.347724 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:49.347830 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:49.347977 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 13:49:49.348032 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:49:49.349013 26105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:49.357061 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:49.356606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36709 (local address 127.0.0.1:33434)
0927 13:49:49.356813 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:49.356903 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 13:49:49.358237 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:49.357834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36709 (local address 127.0.0.1:33446)
0927 13:49:49.358018 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:49.358087 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 13:49:49.437601 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 25983
I20260927 13:49:49.439898 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:49.439339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36709 (local address 127.0.0.1:33460)
0927 13:49:49.439633 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:49.439729 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 13:49:49.441052 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:49.440647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36709 (local address 127.0.0.1:33468)
0927 13:49:49.440842 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:49.440911 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 13:49:49.451025 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 25632
I20260927 13:49:49.452324 25693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:49.560312 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 25632
2026-09-27T13:49:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2918 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 13:49:49.598701 21013 mini_oidc.cc:152] Starting JWKS server
I20260927 13:49:49.601533 21013 webserver.cc:460] Webserver started at https://127.0.0.1:42477/ using document root <none> and password file <none>
I20260927 13:49:49.601887 21013 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 13:49:49.602587 21013 webserver.cc:460] Webserver started at http://127.0.0.1:40365/ using document root <none> and password file <none>
2026-09-27T13:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:49Z Disabled control of system clock
I20260927 13:49:49.630205 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41319
--webserver_interface=127.20.133.126
--webserver_port=0
--builtin_ntp_servers=127.20.133.84:33409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:41319
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42477/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/testchainca.cert with env {}
W20260927 13:49:49.954820 26167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:49.955158 26167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:49.955291 26167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:49.962747 26167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:49.962891 26167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:49.962944 26167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:49.963029 26167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:49.963088 26167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:49:49.963174 26167 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:49:49.963229 26167 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42477/jwks.json
I20260927 13:49:49.974203 26167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:33409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:41319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42477/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:49.975607 26167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:49.976980 26167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:49.989573 26173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:49.989632 26172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:49.989573 26175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:49.991134 26167 server_base.cc:1034] running on GCE node
I20260927 13:49:49.991971 26167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:49.993542 26167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:49.994812 26167 hybrid_clock.cc:648] HybridClock initialized: now 1790516989994738 us; error 58 us; skew 500 ppm
I20260927 13:49:49.995222 26167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:49:49.997741 26167 webserver.cc:460] Webserver started at http://127.20.133.126:39129/ using document root <none> and password file <none>
I20260927 13:49:49.998471 26167 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:49.998592 26167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:49.998893 26167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:50.001942 26167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "3aaab7032f9a4187b6c5b065a29f58b2"
format_stamp: "Formatted at 2026-09-27 13:49:49 on dist-test-slave-q25r"
server_key: "d75efed22f132ad7b0a54fbcf90a9eb9"
server_key_iv: "62fd73d70e58bd052b13736e513c6614"
server_key_version: "encryptionkey@0"
I20260927 13:49:50.002751 26167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "3aaab7032f9a4187b6c5b065a29f58b2"
format_stamp: "Formatted at 2026-09-27 13:49:49 on dist-test-slave-q25r"
server_key: "d75efed22f132ad7b0a54fbcf90a9eb9"
server_key_iv: "62fd73d70e58bd052b13736e513c6614"
server_key_version: "encryptionkey@0"
I20260927 13:49:50.008589 26167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:49:50.012866 26181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:50.014685 26167 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:49:50.014926 26167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "3aaab7032f9a4187b6c5b065a29f58b2"
format_stamp: "Formatted at 2026-09-27 13:49:49 on dist-test-slave-q25r"
server_key: "d75efed22f132ad7b0a54fbcf90a9eb9"
server_key_iv: "62fd73d70e58bd052b13736e513c6614"
server_key_version: "encryptionkey@0"
I20260927 13:49:50.015157 26167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:50.041837 26167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:50.042717 26167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:50.043009 26167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:50.064602 26167 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:41319
I20260927 13:49:50.064631 26232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:41319 every 8 connection(s)
I20260927 13:49:50.066134 26167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:50.070354 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 26167
I20260927 13:49:50.070554 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:50.071137 21013 external_mini_cluster.cc:1428] Setting key fd74d4f8053900fd9a8f6596d320b493
I20260927 13:49:50.071676 26233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:50.089907 26233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2: Bootstrap starting.
I20260927 13:49:50.090379 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:50.073470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.20.133.126:41319)
0927 13:49:50.074805 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:50.074824 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:50.074902 (+ 78us) server_negotiation.cc:407] Connection header received
0927 13:49:50.077650 (+ 2748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:50.077690 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:50.078533 (+ 843us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:50.079162 (+ 629us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:50.079897 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:50.082094 (+ 2197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:50.083759 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:50.084954 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:50.085224 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:50.086761 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:50.086772 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:50.086798 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:50.087694 (+ 896us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:50.087815 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:50.087857 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:50.087987 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:50.088291 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 13:49:50.088759 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":418,"threads_started":1}
I20260927 13:49:50.094099 26233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:50.095381 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:49:50.095381 26233 log.cc:824] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:50.099210 26233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2: No bootstrap required, opened a new log
I20260927 13:49:50.107520 26233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaab7032f9a4187b6c5b065a29f58b2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41319 } }
I20260927 13:49:50.108009 26233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:50.108115 26233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aaab7032f9a4187b6c5b065a29f58b2, State: Initialized, Role: FOLLOWER
I20260927 13:49:50.108908 26233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaab7032f9a4187b6c5b065a29f58b2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41319 } }
I20260927 13:49:50.109179 26233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:50.109323 26233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:50.109520 26233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:50.112001 26233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaab7032f9a4187b6c5b065a29f58b2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41319 } }
I20260927 13:49:50.112685 26233 leader_election.cc:302] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aaab7032f9a4187b6c5b065a29f58b2; no voters:
I20260927 13:49:50.113436 26233 leader_election.cc:288] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:50.113618 26238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:50.114452 26238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [term 1 LEADER]: Becoming Leader. State: Replica: 3aaab7032f9a4187b6c5b065a29f58b2, State: Running, Role: LEADER
I20260927 13:49:50.115097 26238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaab7032f9a4187b6c5b065a29f58b2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41319 } }
I20260927 13:49:50.116016 26233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:50.118569 26240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aaab7032f9a4187b6c5b065a29f58b2. Latest consensus state: current_term: 1 leader_uuid: "3aaab7032f9a4187b6c5b065a29f58b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaab7032f9a4187b6c5b065a29f58b2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41319 } } }
I20260927 13:49:50.119580 26240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:50.118883 26239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aaab7032f9a4187b6c5b065a29f58b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaab7032f9a4187b6c5b065a29f58b2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41319 } } }
I20260927 13:49:50.120201 26239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:50.121973 26243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:50.127007 26243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:50.136698 26243 catalog_manager.cc:1347] Generated new cluster ID: 5b33c4454bf746e49e910b809f1c00ed
I20260927 13:49:50.136845 26243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:50.156867 26243 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:50.159190 26243 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:50.171926 26243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aaab7032f9a4187b6c5b065a29f58b2: Generated new TSK 0
I20260927 13:49:50.173382 26243 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 13:49:50.218585 26263 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 13:49:50.218789 26263 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 13:49:50.219815 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:50.197491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.20.133.126:41319)
0927 13:49:50.197806 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:50.197815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:50.198460 (+ 645us) server_negotiation.cc:407] Connection header received
0927 13:49:50.198882 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:50.198887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:50.198983 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:50.199170 (+ 187us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:49:50.199890 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:50.200730 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:50.202966 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:50.203544 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:50.203656 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:50.204475 (+ 819us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:49:50.219230 (+ 14755us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42477/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 13:49:50.219532 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33164: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42477/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":104}
W20260927 13:49:50.219938 26236 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33164: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42477/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 13:49:50.222452 26262 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:49:50.197407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:41319 (local address 127.0.0.1:33164)
0927 13:49:50.198273 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:50.198298 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:50.198568 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:50.199226 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:50.199231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:50.199247 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:49:50.199717 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:50.199731 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:50.200893 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:50.200899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:50.202769 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:50.202798 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:50.203754 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:50.203762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:50.203974 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:50.204080 (+ 106us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:49:50.219641 (+ 15561us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42477/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 13:49:50.220194 (+ 553us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.133.126:41319: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42477/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":410,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.133.126:41319: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42477/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 13:49:50.224858 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 26167
I20260927 13:49:50.226572 26228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:50.332692 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 26167
2026-09-27T13:49:50Z chronyd exiting
I20260927 13:49:50.383803 21013 test_util.cc:175] -----------------------------------------------
I20260927 13:49:50.383966 21013 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516948056776-21013-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (789 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 13:49:50.388114 21013 mini_oidc.cc:152] Starting JWKS server
I20260927 13:49:50.390183 21013 webserver.cc:460] Webserver started at https://127.0.0.1:38701/ using document root <none> and password file <none>
I20260927 13:49:50.390437 21013 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 13:49:50.391192 21013 webserver.cc:460] Webserver started at http://127.0.0.1:43919/ using document root <none> and password file <none>
2026-09-27T13:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:50Z Disabled control of system clock
I20260927 13:49:50.421223 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:42383
--webserver_interface=127.20.133.126
--webserver_port=0
--builtin_ntp_servers=127.20.133.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:42383
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38701/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/testchainca.cert with env {}
W20260927 13:49:50.788638 26270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:50.788971 26270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:50.789091 26270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:50.797189 26270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:50.797351 26270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:50.797417 26270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:50.797495 26270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:50.797557 26270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:49:50.797652 26270 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:49:50.797708 26270 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38701/jwks.json
I20260927 13:49:50.808606 26270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:42383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38701/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:50.810077 26270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:50.811493 26270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:50.824473 26278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:50.824542 26275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:50.824548 26276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:50.825523 26270 server_base.cc:1034] running on GCE node
I20260927 13:49:50.826411 26270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:50.827966 26270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:50.829236 26270 hybrid_clock.cc:648] HybridClock initialized: now 1790516990829150 us; error 73 us; skew 500 ppm
I20260927 13:49:50.829682 26270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:49:50.832659 26270 webserver.cc:460] Webserver started at http://127.20.133.126:42637/ using document root <none> and password file <none>
I20260927 13:49:50.833492 26270 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:50.833660 26270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:50.834023 26270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:50.837554 26270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "cae1bb4d8dc7464c8a1c7ccde21a26cd"
format_stamp: "Formatted at 2026-09-27 13:49:50 on dist-test-slave-q25r"
server_key: "9e02061cebac7b840ee242c2c12a85e7"
server_key_iv: "8b1d36ee520fe21f21ba9fe416333e62"
server_key_version: "encryptionkey@0"
I20260927 13:49:50.838459 26270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "cae1bb4d8dc7464c8a1c7ccde21a26cd"
format_stamp: "Formatted at 2026-09-27 13:49:50 on dist-test-slave-q25r"
server_key: "9e02061cebac7b840ee242c2c12a85e7"
server_key_iv: "8b1d36ee520fe21f21ba9fe416333e62"
server_key_version: "encryptionkey@0"
I20260927 13:49:50.844726 26270 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 13:49:50.849579 26284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:50.851761 26270 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:49:50.852207 26270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "cae1bb4d8dc7464c8a1c7ccde21a26cd"
format_stamp: "Formatted at 2026-09-27 13:49:50 on dist-test-slave-q25r"
server_key: "9e02061cebac7b840ee242c2c12a85e7"
server_key_iv: "8b1d36ee520fe21f21ba9fe416333e62"
server_key_version: "encryptionkey@0"
I20260927 13:49:50.852653 26270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:50.877416 26270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:50.878324 26270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:50.878641 26270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:50.898888 26270 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:42383
I20260927 13:49:50.898965 26335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:42383 every 8 connection(s)
I20260927 13:49:50.900388 26270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:50.905201 26336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:50.905673 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 26270
I20260927 13:49:50.905900 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:50.906774 21013 external_mini_cluster.cc:1428] Setting key b4282c36c18651ae24c868e8eb00afcd
I20260927 13:49:50.922634 26336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd: Bootstrap starting.
I20260927 13:49:50.923933 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:50.909300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.20.133.126:42383)
0927 13:49:50.910716 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:50.910729 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:50.910786 (+ 57us) server_negotiation.cc:407] Connection header received
0927 13:49:50.913630 (+ 2844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:50.913662 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:50.914298 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:50.914701 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:50.915463 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:50.916874 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:50.918827 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:50.919639 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:50.919858 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:50.921279 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:50.921288 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:50.921305 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:50.921933 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:50.922031 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:50.922060 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:50.922167 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:50.922405 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 13:49:50.922769 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":541,"threads_started":1}
I20260927 13:49:50.926916 26336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:50.928362 26336 log.cc:824] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:50.928740 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:49:50.931897 26336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd: No bootstrap required, opened a new log
I20260927 13:49:50.940069 26336 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae1bb4d8dc7464c8a1c7ccde21a26cd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 42383 } }
I20260927 13:49:50.940615 26336 raft_consensus.cc:374] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:50.940718 26336 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae1bb4d8dc7464c8a1c7ccde21a26cd, State: Initialized, Role: FOLLOWER
I20260927 13:49:50.941377 26336 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae1bb4d8dc7464c8a1c7ccde21a26cd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 42383 } }
I20260927 13:49:50.941636 26336 raft_consensus.cc:388] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:50.941864 26336 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:50.942029 26336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:50.944522 26336 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae1bb4d8dc7464c8a1c7ccde21a26cd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 42383 } }
I20260927 13:49:50.945138 26336 leader_election.cc:302] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae1bb4d8dc7464c8a1c7ccde21a26cd; no voters:
I20260927 13:49:50.945842 26336 leader_election.cc:288] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:50.946079 26341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:50.946923 26341 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [term 1 LEADER]: Becoming Leader. State: Replica: cae1bb4d8dc7464c8a1c7ccde21a26cd, State: Running, Role: LEADER
I20260927 13:49:50.947613 26341 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae1bb4d8dc7464c8a1c7ccde21a26cd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 42383 } }
I20260927 13:49:50.948216 26336 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:50.950656 26343 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae1bb4d8dc7464c8a1c7ccde21a26cd. Latest consensus state: current_term: 1 leader_uuid: "cae1bb4d8dc7464c8a1c7ccde21a26cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae1bb4d8dc7464c8a1c7ccde21a26cd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 42383 } } }
I20260927 13:49:50.951189 26343 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:50.951133 26342 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cae1bb4d8dc7464c8a1c7ccde21a26cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae1bb4d8dc7464c8a1c7ccde21a26cd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 42383 } } }
I20260927 13:49:50.951485 26342 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:50.952294 26349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:50.959332 26349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:50.970225 26349 catalog_manager.cc:1347] Generated new cluster ID: e1cedb40f2af4a63b236c62566b6c191
I20260927 13:49:50.970382 26349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:50.993144 26349 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:50.994618 26349 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:51.005574 26349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cae1bb4d8dc7464c8a1c7ccde21a26cd: Generated new TSK 0
I20260927 13:49:51.007170 26349 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 13:49:51.044736 26366 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 13:49:51.044855 26366 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 13:49:51.046564 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:51.024819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.20.133.126:42383)
0927 13:49:51.025088 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:51.025095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:51.025314 (+ 219us) server_negotiation.cc:407] Connection header received
0927 13:49:51.025583 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:51.025591 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:51.025708 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:51.025942 (+ 234us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:49:51.026481 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:51.027697 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:51.029197 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:51.029933 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:51.030078 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:51.030640 (+ 562us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:49:51.045689 (+ 15049us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38701/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 13:49:51.046193 (+ 504us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43458: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38701/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":102}
W20260927 13:49:51.046602 26365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:49:51.024600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:42383 (local address 127.0.0.1:43458)
0927 13:49:51.025200 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:51.025217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:51.025363 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:51.025987 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:51.025992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:51.026004 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:49:51.026299 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:51.026308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:51.027869 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:51.027873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:51.029042 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:51.029055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:51.030124 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:51.030130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:51.030280 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:51.030319 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:49:51.046035 (+ 15716us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38701/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 13:49:51.046284 (+ 249us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.133.126:42383: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38701/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":311,"threads_started":1}
W20260927 13:49:51.046722 26339 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43458: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38701/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 13:49:51.047724 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 26270
I20260927 13:49:51.049001 26331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:51.156361 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 26270
2026-09-27T13:49:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (823 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 13:49:51.211824 21013 mini_oidc.cc:152] Starting JWKS server
I20260927 13:49:51.213848 21013 webserver.cc:460] Webserver started at https://127.0.0.1:43149/ using document root <none> and password file <none>
I20260927 13:49:51.214105 21013 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 13:49:51.214766 21013 webserver.cc:460] Webserver started at http://127.0.0.1:43403/ using document root <none> and password file <none>
2026-09-27T13:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:51Z Disabled control of system clock
I20260927 13:49:51.243906 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:44241
--webserver_interface=127.20.133.126
--webserver_port=0
--builtin_ntp_servers=127.20.133.84:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:44241
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43149/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 13:49:51.621634 26373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:51.622081 26373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:51.622239 26373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:51.630122 26373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:51.630271 26373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:51.630337 26373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:51.630424 26373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:51.630486 26373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:49:51.630577 26373 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:49:51.630628 26373 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43149/jwks.json
I20260927 13:49:51.640782 26373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:44241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:44241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43149/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:51.642517 26373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:51.644812 26373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:51.658463 26378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:51.658484 26379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:51.658461 26381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:51.660076 26373 server_base.cc:1034] running on GCE node
I20260927 13:49:51.661167 26373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:51.663061 26373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:51.664322 26373 hybrid_clock.cc:648] HybridClock initialized: now 1790516991664271 us; error 43 us; skew 500 ppm
I20260927 13:49:51.664861 26373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:49:51.667959 26373 webserver.cc:460] Webserver started at http://127.20.133.126:44185/ using document root <none> and password file <none>
I20260927 13:49:51.668979 26373 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:51.669083 26373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:51.669459 26373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:51.673152 26373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "33c5ecbc01aa44ada89f5269db18d7fc"
format_stamp: "Formatted at 2026-09-27 13:49:51 on dist-test-slave-q25r"
server_key: "98e570753357f8209327f2c3e33f8a8f"
server_key_iv: "8ef2e344078772b529b5613dd22cd305"
server_key_version: "encryptionkey@0"
I20260927 13:49:51.674218 26373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "33c5ecbc01aa44ada89f5269db18d7fc"
format_stamp: "Formatted at 2026-09-27 13:49:51 on dist-test-slave-q25r"
server_key: "98e570753357f8209327f2c3e33f8a8f"
server_key_iv: "8ef2e344078772b529b5613dd22cd305"
server_key_version: "encryptionkey@0"
I20260927 13:49:51.681092 26373 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 13:49:51.685683 26387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:51.687780 26373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:49:51.688020 26373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "33c5ecbc01aa44ada89f5269db18d7fc"
format_stamp: "Formatted at 2026-09-27 13:49:51 on dist-test-slave-q25r"
server_key: "98e570753357f8209327f2c3e33f8a8f"
server_key_iv: "8ef2e344078772b529b5613dd22cd305"
server_key_version: "encryptionkey@0"
I20260927 13:49:51.688302 26373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:51.718199 26373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:51.719317 26373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:51.719683 26373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:51.741732 26373 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:44241
I20260927 13:49:51.741710 26438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:44241 every 8 connection(s)
I20260927 13:49:51.743388 26373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:51.747764 26439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:51.748462 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 26373
I20260927 13:49:51.748713 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:51.749622 21013 external_mini_cluster.cc:1428] Setting key b2cf5a5f197dd20ab90dd8e9c915a0a5
I20260927 13:49:51.762241 26439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc: Bootstrap starting.
I20260927 13:49:51.764979 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:51.751844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.20.133.126:44241)
0927 13:49:51.753089 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:51.753105 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:51.753151 (+ 46us) server_negotiation.cc:407] Connection header received
0927 13:49:51.755083 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:51.755113 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:51.755751 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:51.756422 (+ 671us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:51.757192 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:51.758540 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:51.760096 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:51.760869 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:51.761051 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:51.762344 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:51.762356 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:51.762376 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:51.762986 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:51.763079 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:51.763103 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:51.763206 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:51.763487 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 13:49:51.763907 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":541,"threads_started":1}
I20260927 13:49:51.765980 26439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:51.767449 26439 log.cc:824] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:51.769709 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:49:51.771000 26439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc: No bootstrap required, opened a new log
I20260927 13:49:51.779547 26439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c5ecbc01aa44ada89f5269db18d7fc" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44241 } }
I20260927 13:49:51.780021 26439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:51.780117 26439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c5ecbc01aa44ada89f5269db18d7fc, State: Initialized, Role: FOLLOWER
I20260927 13:49:51.780794 26439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c5ecbc01aa44ada89f5269db18d7fc" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44241 } }
I20260927 13:49:51.781039 26439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:51.781172 26439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:51.781325 26439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:51.783602 26439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c5ecbc01aa44ada89f5269db18d7fc" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44241 } }
I20260927 13:49:51.784190 26439 leader_election.cc:302] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c5ecbc01aa44ada89f5269db18d7fc; no voters:
I20260927 13:49:51.784899 26439 leader_election.cc:288] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:51.785049 26444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:51.785765 26444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [term 1 LEADER]: Becoming Leader. State: Replica: 33c5ecbc01aa44ada89f5269db18d7fc, State: Running, Role: LEADER
I20260927 13:49:51.786384 26444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c5ecbc01aa44ada89f5269db18d7fc" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44241 } }
I20260927 13:49:51.787386 26439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:51.789389 26445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c5ecbc01aa44ada89f5269db18d7fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c5ecbc01aa44ada89f5269db18d7fc" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44241 } } }
I20260927 13:49:51.789829 26445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:51.790268 26446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c5ecbc01aa44ada89f5269db18d7fc. Latest consensus state: current_term: 1 leader_uuid: "33c5ecbc01aa44ada89f5269db18d7fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c5ecbc01aa44ada89f5269db18d7fc" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44241 } } }
I20260927 13:49:51.790570 26446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:51.792794 26452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:51.798321 26452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:51.808878 26452 catalog_manager.cc:1347] Generated new cluster ID: 39b219ab82c2482f931880180a0341f8
I20260927 13:49:51.809008 26452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:51.843218 26452 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:51.844718 26452 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:51.855504 26452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c5ecbc01aa44ada89f5269db18d7fc: Generated new TSK 0
I20260927 13:49:51.856503 26452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:49:51.885011 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:51.876613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.20.133.126:44241)
0927 13:49:51.876823 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:51.876830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:51.877199 (+ 369us) server_negotiation.cc:407] Connection header received
0927 13:49:51.877439 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:51.877444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:51.877527 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:51.877687 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:49:51.878307 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:51.879069 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:51.881142 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:51.881611 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:51.881714 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:51.882377 (+ 663us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:49:51.884378 (+ 2001us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43149/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 13:49:51.884706 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56984: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43149/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 13:49:51.885144 26442 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56984: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43149/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 13:49:51.885248 26468 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:49:51.876381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:44241 (local address 127.0.0.1:56984)
0927 13:49:51.877034 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:51.877068 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:51.877243 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:51.877714 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:51.877719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:51.877733 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:49:51.878116 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:51.878129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:51.879221 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:51.879227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:51.880951 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:51.880969 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:51.881805 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:51.881812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:51.882000 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:51.882061 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:49:51.884578 (+ 2517us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43149/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 13:49:51.884845 (+ 267us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.133.126:44241: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43149/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":345,"threads_started":1}
I20260927 13:49:51.886123 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 26373
I20260927 13:49:51.887298 26434 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 13:49:51.897070 26469 webserver.cc:533] Webserver: sslize failed:
E20260927 13:49:51.897186 26469 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 13:49:51.980890 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 26373
2026-09-27T13:49:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (821 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T13:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:52Z Disabled control of system clock
I20260927 13:49:52.060211 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:39613
--webserver_interface=127.20.133.126
--webserver_port=0
--builtin_ntp_servers=127.20.133.84:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:39613
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/testchainca.cert with env {}
W20260927 13:49:52.417676 26474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:52.418052 26474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:52.418148 26474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:52.426319 26474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:52.426477 26474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:52.426520 26474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:52.426614 26474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:52.426671 26474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:49:52.426748 26474 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:49:52.426795 26474 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 13:49:52.437513 26474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:39613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:52.438879 26474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:52.440580 26474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:52.453238 26480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:52.453487 26482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:52.454232 26474 server_base.cc:1034] running on GCE node
W20260927 13:49:52.454344 26479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:52.455281 26474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:52.456902 26474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:52.458161 26474 hybrid_clock.cc:648] HybridClock initialized: now 1790516992458106 us; error 52 us; skew 500 ppm
I20260927 13:49:52.458552 26474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:49:52.461043 26474 webserver.cc:460] Webserver started at http://127.20.133.126:36677/ using document root <none> and password file <none>
I20260927 13:49:52.461760 26474 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:52.461867 26474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:52.462163 26474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:52.465171 26474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "cabb160656a04548925900ce9412b3ce"
format_stamp: "Formatted at 2026-09-27 13:49:52 on dist-test-slave-q25r"
server_key: "a0bada1a003389b9018fa0763bf9f66c"
server_key_iv: "7a58c7f5407639b2c19ef44d1c73b0d3"
server_key_version: "encryptionkey@0"
I20260927 13:49:52.465963 26474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "cabb160656a04548925900ce9412b3ce"
format_stamp: "Formatted at 2026-09-27 13:49:52 on dist-test-slave-q25r"
server_key: "a0bada1a003389b9018fa0763bf9f66c"
server_key_iv: "7a58c7f5407639b2c19ef44d1c73b0d3"
server_key_version: "encryptionkey@0"
I20260927 13:49:52.471397 26474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:49:52.475052 26488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:52.476608 26474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:49:52.476794 26474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "cabb160656a04548925900ce9412b3ce"
format_stamp: "Formatted at 2026-09-27 13:49:52 on dist-test-slave-q25r"
server_key: "a0bada1a003389b9018fa0763bf9f66c"
server_key_iv: "7a58c7f5407639b2c19ef44d1c73b0d3"
server_key_version: "encryptionkey@0"
I20260927 13:49:52.477016 26474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:52.507226 26474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:52.508109 26474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:52.508445 26474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:52.526111 26474 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:39613
I20260927 13:49:52.526134 26539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:39613 every 8 connection(s)
I20260927 13:49:52.527554 26474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:52.531778 26540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:52.532521 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 26474
I20260927 13:49:52.532687 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:52.533310 21013 external_mini_cluster.cc:1428] Setting key 8a90f0302a19a3932ba58a5c11d3dc46
I20260927 13:49:52.545060 26540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce: Bootstrap starting.
I20260927 13:49:52.547242 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:52.535329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.20.133.126:39613)
0927 13:49:52.536512 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:52.536533 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:52.536602 (+ 69us) server_negotiation.cc:407] Connection header received
0927 13:49:52.538506 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:52.538534 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:52.539101 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:52.539525 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:52.540096 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:52.541428 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:52.542715 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:52.543455 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:52.543626 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:52.544887 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:52.544896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:52.544911 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:49:52.545532 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:52.545628 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:49:52.545651 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:52.545757 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:52.545965 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 13:49:52.546294 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":369,"threads_started":1}
I20260927 13:49:52.548626 26540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:52.549717 26540 log.cc:824] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:52.551059 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:49:52.552435 26540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce: No bootstrap required, opened a new log
I20260927 13:49:52.559603 26540 raft_consensus.cc:348] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabb160656a04548925900ce9412b3ce" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39613 } }
I20260927 13:49:52.560009 26540 raft_consensus.cc:374] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:52.560118 26540 raft_consensus.cc:729] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabb160656a04548925900ce9412b3ce, State: Initialized, Role: FOLLOWER
I20260927 13:49:52.560763 26540 consensus_queue.cc:260] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabb160656a04548925900ce9412b3ce" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39613 } }
I20260927 13:49:52.560981 26540 raft_consensus.cc:388] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:52.561084 26540 raft_consensus.cc:482] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:52.561213 26540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:52.563395 26540 raft_consensus.cc:504] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabb160656a04548925900ce9412b3ce" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39613 } }
I20260927 13:49:52.563934 26540 leader_election.cc:302] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cabb160656a04548925900ce9412b3ce; no voters:
I20260927 13:49:52.564563 26540 leader_election.cc:288] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:52.564710 26545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:52.565765 26545 raft_consensus.cc:686] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [term 1 LEADER]: Becoming Leader. State: Replica: cabb160656a04548925900ce9412b3ce, State: Running, Role: LEADER
I20260927 13:49:52.566304 26545 consensus_queue.cc:237] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabb160656a04548925900ce9412b3ce" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39613 } }
I20260927 13:49:52.566771 26540 sys_catalog.cc:564] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:52.569722 26547 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader cabb160656a04548925900ce9412b3ce. Latest consensus state: current_term: 1 leader_uuid: "cabb160656a04548925900ce9412b3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabb160656a04548925900ce9412b3ce" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39613 } } }
I20260927 13:49:52.570271 26547 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:52.570152 26546 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cabb160656a04548925900ce9412b3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabb160656a04548925900ce9412b3ce" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39613 } } }
I20260927 13:49:52.570611 26546 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:52.572047 26554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:52.577471 26554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:52.586261 26554 catalog_manager.cc:1347] Generated new cluster ID: c0c65fee70fe4c798324b5a86e880487
I20260927 13:49:52.586401 26554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:52.598613 26554 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:52.600083 26554 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:52.608767 26554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cabb160656a04548925900ce9412b3ce: Generated new TSK 0
I20260927 13:49:52.609673 26554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:49:53.108543 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:52.626036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.20.133.126:39613)
0927 13:49:52.626257 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:52.626263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:52.626514 (+ 251us) server_negotiation.cc:407] Connection header received
0927 13:49:52.626732 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:52.626738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:52.626836 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:52.627018 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:49:52.627456 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:52.628619 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:52.629978 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:52.630637 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:52.630762 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:52.631214 (+ 452us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:49:53.107868 (+476654us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 13:49:53.108222 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55292: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 13:49:53.108661 26543 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55292: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 13:49:53.109213 26569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:49:52.625876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:39613 (local address 127.0.0.1:55292)
0927 13:49:52.626424 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:52.626440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:52.626555 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:52.627046 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:52.627050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:52.627060 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:49:52.627306 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:52.627314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:52.628746 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:52.628750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:52.629829 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:52.629840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:52.630796 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:52.630801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:52.630915 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:52.630948 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:49:53.108329 (+477381us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 13:49:53.108753 (+ 424us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.133.126:39613: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":286,"threads_started":1}
I20260927 13:49:53.110903 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 26474
I20260927 13:49:53.112574 26535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:53.229187 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 26474
2026-09-27T13:49:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1229 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (483 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (432 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:49:54 dist-test-slave-q25r krb5kdc[26584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:49:54 dist-test-slave-q25r krb5kdc[26584](info): set up 2 sockets
Sep 27 13:49:54 dist-test-slave-q25r krb5kdc[26584](info): commencing operation
krb5kdc: starting...
W20260927 13:49:55.157291 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:49:55 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:49:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:49:55.404452 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:38509
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:38509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:55.776512 26597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:55.776885 26597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:55.777010 26597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:55.785266 26597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:55.785431 26597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:55.785498 26597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:55.785586 26597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:55.785635 26597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:55.797029 26597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:38509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:38509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:55.798846 26597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:55.800503 26597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:55.814543 26602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:55.814623 26603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:55.814599 26605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:55.815655 26597 server_base.cc:1034] running on GCE node
I20260927 13:49:55.816579 26597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:55.818235 26597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:55.819492 26597 hybrid_clock.cc:648] HybridClock initialized: now 1790516995819416 us; error 69 us; skew 500 ppm
I20260927 13:49:55.819882 26597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:55 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516995, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:55.823863 26597 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:49:55.825996 26597 webserver.cc:460] Webserver started at http://127.20.133.126:39679/ using document root <none> and password file <none>
I20260927 13:49:55.826879 26597 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:55.827014 26597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:55.827375 26597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:55.830948 26597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "9082383e2e9240bea6038448b0e30271"
format_stamp: "Formatted at 2026-09-27 13:49:55 on dist-test-slave-q25r"
server_key: "51feb6939678e5c04018630802b9c0ac"
server_key_iv: "375990c6fa50f5e1a2904112d9d23968"
server_key_version: "encryptionkey@0"
I20260927 13:49:55.831815 26597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "9082383e2e9240bea6038448b0e30271"
format_stamp: "Formatted at 2026-09-27 13:49:55 on dist-test-slave-q25r"
server_key: "51feb6939678e5c04018630802b9c0ac"
server_key_iv: "375990c6fa50f5e1a2904112d9d23968"
server_key_version: "encryptionkey@0"
I20260927 13:49:55.838105 26597 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 13:49:55.842276 26612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:55.843788 26597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:49:55.843981 26597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "9082383e2e9240bea6038448b0e30271"
format_stamp: "Formatted at 2026-09-27 13:49:55 on dist-test-slave-q25r"
server_key: "51feb6939678e5c04018630802b9c0ac"
server_key_iv: "375990c6fa50f5e1a2904112d9d23968"
server_key_version: "encryptionkey@0"
I20260927 13:49:55.844203 26597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:55.868772 26597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:55.873289 26597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:55.873601 26597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:55.893309 26597 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:38509
I20260927 13:49:55.893333 26663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:38509 every 8 connection(s)
I20260927 13:49:55.894876 26597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:55.898783 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 26597
I20260927 13:49:55.898986 26664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:49:55.899008 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:55.899803 21013 external_mini_cluster.cc:1428] Setting key 7bd49cb9bc52cfea6a3249222893ea86
I20260927 13:49:55.914080 26664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271: Bootstrap starting.
Sep 27 13:49:55 dist-test-slave-q25r krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:55.917919 26664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:55.919063 26664 log.cc:824] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:55.922184 26664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271: No bootstrap required, opened a new log
I20260927 13:49:55.923535 26667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:55.901915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43604 (local address 127.20.133.126:38509)
0927 13:49:55.902746 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:55.902758 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:55.902800 (+ 42us) server_negotiation.cc:407] Connection header received
0927 13:49:55.904412 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:55.904435 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:55.904849 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:55.905201 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:55.907035 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:55.908376 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:55.910101 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:55.910857 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:55.911040 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:55.915799 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:55.915808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:55.915824 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:55.915882 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:55.919128 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:55.919942 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:55.919951 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:55.919961 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:55.920189 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:55.920632 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:55.920637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:55.920640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:55.921225 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:55.921531 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:55.921967 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 13:49:55.922419 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":375,"threads_started":1}
I20260927 13:49:55.930147 26664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9082383e2e9240bea6038448b0e30271" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38509 } }
I20260927 13:49:55.930562 26664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:55.930650 26664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9082383e2e9240bea6038448b0e30271, State: Initialized, Role: FOLLOWER
I20260927 13:49:55.931221 26664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9082383e2e9240bea6038448b0e30271" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38509 } }
I20260927 13:49:55.931452 26664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:55.931572 26664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:55.931718 26664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:55.934044 26664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9082383e2e9240bea6038448b0e30271" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38509 } }
I20260927 13:49:55.934561 26664 leader_election.cc:302] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9082383e2e9240bea6038448b0e30271; no voters:
I20260927 13:49:55.935133 26664 leader_election.cc:288] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:55.935295 26669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:55.935920 26669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [term 1 LEADER]: Becoming Leader. State: Replica: 9082383e2e9240bea6038448b0e30271, State: Running, Role: LEADER
I20260927 13:49:55.936515 26669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9082383e2e9240bea6038448b0e30271" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38509 } }
I20260927 13:49:55.937282 26664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:55.939369 26671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9082383e2e9240bea6038448b0e30271. Latest consensus state: current_term: 1 leader_uuid: "9082383e2e9240bea6038448b0e30271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9082383e2e9240bea6038448b0e30271" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38509 } } }
I20260927 13:49:55.939520 26670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9082383e2e9240bea6038448b0e30271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9082383e2e9240bea6038448b0e30271" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38509 } } }
I20260927 13:49:55.939771 26671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:55.939836 26670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:55.941721 26678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:55.947014 26678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:55.956614 26678 catalog_manager.cc:1347] Generated new cluster ID: 4bbc6b1f9d2f49a3adfc52246e56f01b
I20260927 13:49:55.956755 26678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:56.011452 26678 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:56.012961 26678 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:56.020925 26678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9082383e2e9240bea6038448b0e30271: Generated new TSK 0
I20260927 13:49:56.022045 26678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:49:56.079980 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38509
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:56.435215 26690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:56.435514 26690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:56.435608 26690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:56.443158 26690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:56.443271 26690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:56.443410 26690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:49:56.453419 26690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:56.454661 26690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:56.455991 26690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:56.468631 26695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:56.468523 26698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:56.468578 26696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:56.469504 26690 server_base.cc:1034] running on GCE node
I20260927 13:49:56.470090 26690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:56.470999 26690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:56.472203 26690 hybrid_clock.cc:648] HybridClock initialized: now 1790516996472136 us; error 60 us; skew 500 ppm
I20260927 13:49:56.472541 26690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:56 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:56.475574 26690 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:49:56.477082 26690 webserver.cc:460] Webserver started at http://127.20.133.65:37203/ using document root <none> and password file <none>
I20260927 13:49:56.477751 26690 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:56.477864 26690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:56.478140 26690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:56.481031 26690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "3999f536b469447bb7db65e11bdde336"
format_stamp: "Formatted at 2026-09-27 13:49:56 on dist-test-slave-q25r"
server_key: "ba28e069369f074e90e7461b8f0d7a24"
server_key_iv: "b71d72925a1d7d6dbedbc295b9e3bc6e"
server_key_version: "encryptionkey@0"
I20260927 13:49:56.481792 26690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "3999f536b469447bb7db65e11bdde336"
format_stamp: "Formatted at 2026-09-27 13:49:56 on dist-test-slave-q25r"
server_key: "ba28e069369f074e90e7461b8f0d7a24"
server_key_iv: "b71d72925a1d7d6dbedbc295b9e3bc6e"
server_key_version: "encryptionkey@0"
I20260927 13:49:56.486967 26690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:49:56.490538 26705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:56.491911 26690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:49:56.492071 26690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "3999f536b469447bb7db65e11bdde336"
format_stamp: "Formatted at 2026-09-27 13:49:56 on dist-test-slave-q25r"
server_key: "ba28e069369f074e90e7461b8f0d7a24"
server_key_iv: "b71d72925a1d7d6dbedbc295b9e3bc6e"
server_key_version: "encryptionkey@0"
I20260927 13:49:56.492265 26690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:56.510567 26690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:56.515081 26690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:56.515332 26690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:56.516255 26690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:56.517652 26690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:56.517722 26690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:56.517813 26690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:56.517855 26690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:56.550715 26690 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:33335
I20260927 13:49:56.550761 26817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:33335 every 8 connection(s)
I20260927 13:49:56.551931 26690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:49:56.554066 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 26690
I20260927 13:49:56.554271 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:49:56.554957 21013 external_mini_cluster.cc:1428] Setting key 9002ca431cb52d64bacd6c31a527500e
Sep 27 13:49:56 dist-test-slave-q25r krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:56.573940 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:56.555390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:34005 (local address 127.20.133.126:38509)
0927 13:49:56.555873 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:56.555880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:56.556966 (+ 1086us) server_negotiation.cc:407] Connection header received
0927 13:49:56.558683 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:56.558689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:56.558779 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:56.558892 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:56.560437 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:56.561421 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:56.563125 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:56.563607 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:56.563734 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:56.567508 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:56.567514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:56.567518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:56.567558 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:56.570629 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:56.571429 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:56.571436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:56.571440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:56.571626 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:56.572052 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:56.572057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:56.572060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:56.572463 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:56.572616 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:56.573311 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 13:49:56.573550 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":259,"threads_started":1}
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
I20260927 13:49:56.574666 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:56.555760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38509 (local address 127.20.133.65:34005)
0927 13:49:56.556648 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:56.556694 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:56.558393 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:56.559089 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:56.559100 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:56.559652 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:56.560243 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:56.560262 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:56.561625 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:56.561629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:56.562949 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:56.562962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:56.563880 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:56.563886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:56.564151 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:56.564909 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:56.564941 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:56.567243 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:56.570789 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:56.570797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:56.570811 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:56.571277 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:56.571738 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:56.571741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:56.571743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:56.571936 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:56.572575 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:56.572582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:56.572923 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 13:49:56.573153 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 13:49:56.573557 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":351,"threads_started":1}
I20260927 13:49:56.576144 26818 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
I20260927 13:49:56.576496 26818 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:56.577241 26818 heartbeater.cc:511] Master 127.20.133.126:38509 requested a full tablet report, sending...
I20260927 13:49:56.579517 26629 ts_manager.cc:194] Registered new tserver with Master: 3999f536b469447bb7db65e11bdde336 (127.20.133.65:33335)
I20260927 13:49:56.581434 26629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:34005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:49:56.603704 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38509
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:56.925822 26824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:56.926249 26824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:56.926381 26824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:56.933885 26824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:56.934006 26824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:56.934119 26824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:49:56.944380 26824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:56.945613 26824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:56.946828 26824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:56.959805 26830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:56.959734 26829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:56.959779 26832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:56.966172 26824 server_base.cc:1034] running on GCE node
I20260927 13:49:56.966882 26824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:56.967916 26824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:56.969139 26824 hybrid_clock.cc:648] HybridClock initialized: now 1790516996969073 us; error 46 us; skew 500 ppm
I20260927 13:49:56.969470 26824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:56 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:56.972568 26824 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:49:56.974095 26824 webserver.cc:460] Webserver started at http://127.20.133.66:46763/ using document root <none> and password file <none>
I20260927 13:49:56.974792 26824 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:56.974906 26824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:56.975199 26824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:56.978062 26824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "ea0633813beb4be089749f60b7c2ef11"
format_stamp: "Formatted at 2026-09-27 13:49:56 on dist-test-slave-q25r"
server_key: "453572a99c0ebe6df0e97f2dbd9ae176"
server_key_iv: "d40436fa0827e4232d07b3ec87436bfc"
server_key_version: "encryptionkey@0"
I20260927 13:49:56.978812 26824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "ea0633813beb4be089749f60b7c2ef11"
format_stamp: "Formatted at 2026-09-27 13:49:56 on dist-test-slave-q25r"
server_key: "453572a99c0ebe6df0e97f2dbd9ae176"
server_key_iv: "d40436fa0827e4232d07b3ec87436bfc"
server_key_version: "encryptionkey@0"
I20260927 13:49:56.983872 26824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 13:49:56.987432 26839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:56.988889 26824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:49:56.989059 26824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "ea0633813beb4be089749f60b7c2ef11"
format_stamp: "Formatted at 2026-09-27 13:49:56 on dist-test-slave-q25r"
server_key: "453572a99c0ebe6df0e97f2dbd9ae176"
server_key_iv: "d40436fa0827e4232d07b3ec87436bfc"
server_key_version: "encryptionkey@0"
I20260927 13:49:56.989277 26824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:57.009613 26824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:57.014183 26824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:57.014477 26824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:57.015502 26824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:57.016953 26824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:57.017035 26824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:57.017172 26824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:57.017225 26824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:57.046950 26824 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:44403
I20260927 13:49:57.046998 26951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:44403 every 8 connection(s)
I20260927 13:49:57.048198 26824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:49:57.054140 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 26824
I20260927 13:49:57.054322 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:49:57.054850 21013 external_mini_cluster.cc:1428] Setting key 6f1f5883b6249447dac3550797b0cb5c
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:57.070007 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:57.050645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:58311 (local address 127.20.133.126:38509)
0927 13:49:57.050863 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:57.050868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:57.052010 (+ 1142us) server_negotiation.cc:407] Connection header received
0927 13:49:57.053893 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:57.053897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:57.053970 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:57.054059 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:57.055986 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.057000 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.059249 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.059674 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.059763 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:57.063669 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:57.063674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:57.063677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:57.063711 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:57.066875 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.067609 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.067615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.067619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.067770 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.068227 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.068231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.068234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.068598 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:57.068743 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:57.069440 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 13:49:57.069683 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 13:49:57.070917 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:57.051106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38509 (local address 127.20.133.66:58311)
0927 13:49:57.051821 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:57.051866 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:57.053588 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:57.054332 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:57.054344 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:57.055071 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:57.055803 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:57.055828 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.057160 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.057165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:57.059122 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:57.059137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.059883 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.059889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:57.060153 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:57.060849 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:57.060884 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:57.063422 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:57.067037 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:57.067044 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:57.067059 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:57.067495 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:57.067911 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:57.067915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:57.067917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:57.068117 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:57.068729 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:57.068736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:57.069094 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 13:49:57.069338 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 13:49:57.069771 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":277,"threads_started":1}
I20260927 13:49:57.072278 26952 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38509
I20260927 13:49:57.072604 26952 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:57.073320 26952 heartbeater.cc:511] Master 127.20.133.126:38509 requested a full tablet report, sending...
I20260927 13:49:57.074784 26629 ts_manager.cc:194] Registered new tserver with Master: ea0633813beb4be089749f60b7c2ef11 (127.20.133.66:44403)
I20260927 13:49:57.075693 26629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:58311
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:49:57.103030 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38509
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:57.418763 26957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:57.419052 26957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:57.419145 26957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:57.426908 26957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:57.427027 26957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:57.427141 26957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:49:57.437005 26957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:57.438258 26957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:57.439519 26957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:57.452244 26962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:57.452255 26963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:57.452435 26965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:57.453923 26957 server_base.cc:1034] running on GCE node
I20260927 13:49:57.454483 26957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:57.455401 26957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:57.456629 26957 hybrid_clock.cc:648] HybridClock initialized: now 1790516997456568 us; error 53 us; skew 500 ppm
I20260927 13:49:57.456933 26957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:57.459884 26957 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:49:57.461298 26957 webserver.cc:460] Webserver started at http://127.20.133.67:43181/ using document root <none> and password file <none>
I20260927 13:49:57.461952 26957 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:57.462064 26957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:57.462353 26957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:57.465202 26957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "020b46a3c44e47cc9f9b23e7cd9ae873"
format_stamp: "Formatted at 2026-09-27 13:49:57 on dist-test-slave-q25r"
server_key: "dacc237755a6e819ddae5618c452e481"
server_key_iv: "d18ce279ed225e882e96c54509d302cb"
server_key_version: "encryptionkey@0"
I20260927 13:49:57.465937 26957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "020b46a3c44e47cc9f9b23e7cd9ae873"
format_stamp: "Formatted at 2026-09-27 13:49:57 on dist-test-slave-q25r"
server_key: "dacc237755a6e819ddae5618c452e481"
server_key_iv: "d18ce279ed225e882e96c54509d302cb"
server_key_version: "encryptionkey@0"
I20260927 13:49:57.471096 26957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 13:49:57.474766 26972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:57.476114 26957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 13:49:57.476272 26957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "020b46a3c44e47cc9f9b23e7cd9ae873"
format_stamp: "Formatted at 2026-09-27 13:49:57 on dist-test-slave-q25r"
server_key: "dacc237755a6e819ddae5618c452e481"
server_key_iv: "d18ce279ed225e882e96c54509d302cb"
server_key_version: "encryptionkey@0"
I20260927 13:49:57.476506 26957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:57.508435 26957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:57.512866 26957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:57.513142 26957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:57.514127 26957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:49:57.515473 26957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:49:57.515550 26957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:57.515669 26957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:49:57.515729 26957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:57.546834 26957 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:45413
I20260927 13:49:57.546880 27084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:45413 every 8 connection(s)
I20260927 13:49:57.548166 26957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:49:57.553107 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 26957
I20260927 13:49:57.553301 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:49:57.553849 21013 external_mini_cluster.cc:1428] Setting key f0e6095d7f8cc233f7847c32ee78ceab
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:49:57.570797 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:57.550861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:37637 (local address 127.20.133.126:38509)
0927 13:49:57.551023 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:57.551027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:57.552499 (+ 1472us) server_negotiation.cc:407] Connection header received
0927 13:49:57.554899 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:57.554906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:57.554997 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:57.555106 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:57.556978 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.557891 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.559424 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.559875 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.559997 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:57.563398 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:57.563403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:57.563407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:57.563443 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:57.566431 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.567273 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.567277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.567279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.567423 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.568296 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.568301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.568304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.568703 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:57.568853 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:57.570193 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0927 13:49:57.570466 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:49:57.571806 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:57.551477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38509 (local address 127.20.133.67:37637)
0927 13:49:57.552305 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 13:49:57.552380 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 13:49:57.554549 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:49:57.555373 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:49:57.555384 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:49:57.555978 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:49:57.556760 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:57.556799 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.558033 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.558037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:57.559262 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:49:57.559277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.560071 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.560076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:49:57.560287 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:57.560966 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:49:57.560988 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:49:57.563135 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:49:57.566629 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:57.566638 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:57.566653 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:57.567131 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:57.567875 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:49:57.567880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:49:57.567883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:49:57.568150 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:49:57.568951 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:49:57.568964 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:49:57.569709 (+ 745us) client_negotiation.cc:769] Sending connection context
0927 13:49:57.570039 (+ 330us) client_negotiation.cc:240] Negotiation successful
0927 13:49:57.570634 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":378,"threads_started":1}
I20260927 13:49:57.573144 27085 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38509
I20260927 13:49:57.573459 27085 heartbeater.cc:464] Registering TS with master...
I20260927 13:49:57.574204 27085 heartbeater.cc:511] Master 127.20.133.126:38509 requested a full tablet report, sending...
I20260927 13:49:57.575796 26629 ts_manager.cc:194] Registered new tserver with Master: 020b46a3c44e47cc9f9b23e7cd9ae873 (127.20.133.67:45413)
I20260927 13:49:57.576763 26629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:37637
I20260927 13:49:57.582816 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:49:57.584965 26818 heartbeater.cc:503] Master 127.20.133.126:38509 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/test-admin.keytab.
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:57.610617 21013 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 13:49:57.615711 27092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:49:57.613233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.20.133.65:33335)
0927 13:49:57.613867 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:57.613877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:57.613911 (+ 34us) server_negotiation.cc:407] Connection header received
0927 13:49:57.613985 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:57.613992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:57.614216 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:57.614355 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:57.615298 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42284: BlockingRecv error: recv got EOF from 127.0.0.1:42284 (error 108)
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":289,"threads_started":1}
W20260927 13:49:57.615880 21013 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/krb5cc' of type FILE
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:57.617579 21013 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:57.619000 21013 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:57.634871 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:57.620700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.20.133.65:33335)
0927 13:49:57.620904 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:57.620910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:57.621169 (+ 259us) server_negotiation.cc:407] Connection header received
0927 13:49:57.621283 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:57.621287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:57.621356 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:57.621444 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:57.622499 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.623681 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.625025 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.625615 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.625731 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:57.629612 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:57.629628 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:57.629644 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:57.629684 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:57.632289 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.633042 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.633049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.633051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.633174 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.633490 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.633493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.633495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.633859 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:57.634080 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:57.634368 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 13:49:57.634580 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 13:49:57.640892 27092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:49:57.638999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.20.133.65:33335)
0927 13:49:57.639204 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:57.639211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:57.639419 (+ 208us) server_negotiation.cc:407] Connection header received
0927 13:49:57.639516 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:57.639519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:57.639608 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:57.639695 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:57.640619 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42304: BlockingRecv error: recv got EOF from 127.0.0.1:42304 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 13:49:57.641225 21013 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/krb5cc' of type FILE
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:57.642813 21013 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:57.644141 21013 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:57.658125 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:57.645617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.20.133.65:33335)
0927 13:49:57.645856 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:57.645863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:57.645973 (+ 110us) server_negotiation.cc:407] Connection header received
0927 13:49:57.646131 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:57.646135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:57.646197 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:57.646274 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:57.647466 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.648216 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.649534 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.649937 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.650024 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:57.653418 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:57.653423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:57.653427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:57.653476 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:57.655901 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.656588 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.656591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.656594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.656732 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.657016 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.657019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.657021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.657281 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:57.657505 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:57.657744 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 13:49:57.657905 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 13:49:57.662213 27092 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:49:57.660515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.20.133.65:33335)
0927 13:49:57.660731 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:57.660737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:57.660994 (+ 257us) server_negotiation.cc:407] Connection header received
0927 13:49:57.661142 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:57.661147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:57.661219 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:57.661298 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:57.661982 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42318: BlockingRecv error: recv got EOF from 127.0.0.1:42318 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 13:49:57.662354 21013 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516948056776-21013-0/krb5kdc/krb5cc' of type FILE
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:57.663810 21013 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:57.665005 21013 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:49:57.680045 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:57.666375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.20.133.65:33335)
0927 13:49:57.666549 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:57.666555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:57.666705 (+ 150us) server_negotiation.cc:407] Connection header received
0927 13:49:57.666833 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:57.666836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:57.666893 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:57.666964 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:57.667839 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.668616 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.670049 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.670777 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.670887 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:57.674515 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:57.674521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:57.674525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:57.674566 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:57.677244 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.677935 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.677939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.677942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.678099 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.678520 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.678526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.678530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.678838 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:57.679084 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:57.679452 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 13:49:57.679747 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:49:57.682185 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 26690
I20260927 13:49:57.694983 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:57.683093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42340 (local address 127.20.133.65:33335)
0927 13:49:57.683350 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:57.683358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:57.683699 (+ 341us) server_negotiation.cc:407] Connection header received
0927 13:49:57.683903 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:57.683908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:57.683979 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:57.684080 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:57.685238 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.685948 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.687259 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.687700 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.687839 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:57.690085 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:57.690090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:57.690094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:57.690130 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:57.692406 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.693099 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.693105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.693108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.693253 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.693609 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.693614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.693618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.693897 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:57.694157 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:57.694473 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 13:49:57.694682 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 13:49:57.695688 26813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:57.778147 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 26690
I20260927 13:49:57.791481 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 26824
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:49:57.806707 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:57.792481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.20.133.66:44403)
0927 13:49:57.793047 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:57.793054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:57.793080 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:49:57.793140 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:57.793146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:57.793351 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:57.793523 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:57.794475 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.795458 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.796788 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.797320 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.797424 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:57.801175 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:57.801188 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:57.801202 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:57.801244 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:57.803903 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.804725 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.804733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.804735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.804858 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.805210 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.805214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.805216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.805553 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:57.805847 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:57.806149 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 13:49:57.806398 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":304,"threads_started":1}
I20260927 13:49:57.807960 26947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:57.890343 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 26824
I20260927 13:49:57.902796 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 26957
Sep 27 13:49:57 dist-test-slave-q25r krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:49:57.918326 27107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:57.903744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.20.133.67:45413)
0927 13:49:57.904382 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:57.904389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:57.904424 (+ 35us) server_negotiation.cc:407] Connection header received
0927 13:49:57.904487 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:57.904494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:57.904702 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:57.904865 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:57.905907 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.906921 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.908125 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:57.908776 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:57.908891 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:57.912875 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:57.912889 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:57.912904 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:57.912943 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:57.915531 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.916237 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.916245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.916247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.916420 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:57.916793 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:57.916798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:57.916801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:57.917156 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:57.917449 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:57.917756 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 13:49:57.918029 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":314,"threads_started":1}
I20260927 13:49:57.919571 27080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:58.001938 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 26957
I20260927 13:49:58.014454 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 26597
I20260927 13:49:58.015733 26659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:49:58.120386 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 26597
2026-09-27T13:49:58Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3978 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:49:58 dist-test-slave-q25r krb5kdc[27112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:49:58 dist-test-slave-q25r krb5kdc[27112](info): set up 2 sockets
Sep 27 13:49:58 dist-test-slave-q25r krb5kdc[27112](info): commencing operation
krb5kdc: starting...
W20260927 13:49:59.141108 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:49:59 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:49:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.126.keytab.
Entry for principal oryx/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.126.keytab.
I20260927 13:49:59.385216 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:33783
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.126.keytab
--principal=oryx/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:33783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:49:59.715123 27125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:49:59.715469 27125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:49:59.715588 27125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:49:59.723176 27125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:49:59.723328 27125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:49:59.723388 27125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:49:59.723479 27125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:49:59.723536 27125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:49:59.734043 27125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:33783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.126.keytab
--principal=oryx/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:33783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:49:59.735471 27125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:49:59.736927 27125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:49:59.749529 27130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:59.750247 27131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:49:59.750203 27133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:49:59.751634 27125 server_base.cc:1034] running on GCE node
I20260927 13:49:59.752933 27125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:49:59.755123 27125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:49:59.756475 27125 hybrid_clock.cc:648] HybridClock initialized: now 1790516999756420 us; error 58 us; skew 500 ppm
I20260927 13:49:59.756913 27125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:49:59 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516999, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:49:59.768309 27125 init.cc:377] Logged in from keytab as oryx/127.20.133.126@KRBTEST.COM (short username oryx)
I20260927 13:49:59.770256 27125 webserver.cc:460] Webserver started at http://127.20.133.126:46747/ using document root <none> and password file <none>
I20260927 13:49:59.770999 27125 fs_manager.cc:359] Metadata directory not provided
I20260927 13:49:59.771119 27125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:49:59.771451 27125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:49:59.774664 27125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "25826f702bd249f280d27611d1ce1ad2"
format_stamp: "Formatted at 2026-09-27 13:49:59 on dist-test-slave-q25r"
server_key: "1f7a868f9360e54184d5bf587ad67d34"
server_key_iv: "eb13613c36c987682eaea7e6d7d2e1f5"
server_key_version: "encryptionkey@0"
I20260927 13:49:59.775483 27125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "25826f702bd249f280d27611d1ce1ad2"
format_stamp: "Formatted at 2026-09-27 13:49:59 on dist-test-slave-q25r"
server_key: "1f7a868f9360e54184d5bf587ad67d34"
server_key_iv: "eb13613c36c987682eaea7e6d7d2e1f5"
server_key_version: "encryptionkey@0"
I20260927 13:49:59.782564 27125 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260927 13:49:59.787034 27140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:49:59.788951 27125 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 13:49:59.789189 27125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "25826f702bd249f280d27611d1ce1ad2"
format_stamp: "Formatted at 2026-09-27 13:49:59 on dist-test-slave-q25r"
server_key: "1f7a868f9360e54184d5bf587ad67d34"
server_key_iv: "eb13613c36c987682eaea7e6d7d2e1f5"
server_key_version: "encryptionkey@0"
I20260927 13:49:59.789417 27125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:49:59.812130 27125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:49:59.816568 27125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:49:59.816887 27125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:49:59.836796 27125 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:33783
I20260927 13:49:59.836851 27191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:33783 every 8 connection(s)
I20260927 13:49:59.838275 27125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:49:59.839318 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 27125
I20260927 13:49:59.839555 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:49:59.840265 21013 external_mini_cluster.cc:1428] Setting key 3550aca5b94acf6baeff957250fc571e
I20260927 13:49:59.842756 27192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:49:59 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:49:59.856905 27192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2: Bootstrap starting.
I20260927 13:49:59.860900 27192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2: Neither blocks nor log segments found. Creating new log.
I20260927 13:49:59.862280 27192 log.cc:824] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2: Log is configured to *not* fsync() on all Append() calls
I20260927 13:49:59.866487 27192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2: No bootstrap required, opened a new log
I20260927 13:49:59.867533 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:49:59.842556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.20.133.126:33783)
0927 13:49:59.843654 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0927 13:49:59.843671 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 13:49:59.843724 (+ 53us) server_negotiation.cc:407] Connection header received
0927 13:49:59.845686 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:49:59.845713 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:49:59.846234 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:49:59.846715 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:49:59.848786 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:59.850917 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:59.852395 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:49:59.853503 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:49:59.853819 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:49:59.858256 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:49:59.858268 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:49:59.858290 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:49:59.858361 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:49:59.862455 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:59.863473 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:59.863481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:59.863487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:59.863682 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:49:59.864129 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:49:59.864133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:49:59.864136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:49:59.864921 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:49:59.865237 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 13:49:59.865682 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 13:49:59.866185 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":505,"threads_started":1}
I20260927 13:49:59.874941 27192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } }
I20260927 13:49:59.875488 27192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:49:59.875612 27192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25826f702bd249f280d27611d1ce1ad2, State: Initialized, Role: FOLLOWER
I20260927 13:49:59.876569 27192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } }
I20260927 13:49:59.876832 27192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:49:59.876945 27192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:49:59.877082 27192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:49:59.879335 27192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } }
I20260927 13:49:59.879860 27192 leader_election.cc:302] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25826f702bd249f280d27611d1ce1ad2; no voters:
I20260927 13:49:59.880589 27192 leader_election.cc:288] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:49:59.880807 27197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:49:59.881990 27197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 1 LEADER]: Becoming Leader. State: Replica: 25826f702bd249f280d27611d1ce1ad2, State: Running, Role: LEADER
I20260927 13:49:59.882781 27197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } }
I20260927 13:49:59.883083 27192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:49:59.886619 27199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25826f702bd249f280d27611d1ce1ad2. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } }
I20260927 13:49:59.886734 27198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } }
I20260927 13:49:59.887087 27199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:59.887176 27198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:49:59.888098 27206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:49:59.894490 27206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:49:59.908357 27206 catalog_manager.cc:1347] Generated new cluster ID: a47906cfaa1d4b80bc3aac7116ef0a20
I20260927 13:49:59.908499 27206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:49:59.929852 27206 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:49:59.932148 27206 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:49:59.943971 27206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2: Generated new TSK 0
I20260927 13:49:59.944895 27206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab.
Entry for principal oryx/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab.
I20260927 13:50:00.005721 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:33783
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab
--principal=oryx/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:00.350600 27218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:00.351083 27218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:00.351255 27218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:00.359849 27218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:00.360009 27218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:00.360167 27218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:50:00.373476 27218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab
--principal=oryx/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:33783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:00.374898 27218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:00.376394 27218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:00.391258 27223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:00.391171 27224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:00.391222 27226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:00.392249 27218 server_base.cc:1034] running on GCE node
I20260927 13:50:00.392969 27218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:00.394170 27218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:00.395525 27218 hybrid_clock.cc:648] HybridClock initialized: now 1790517000395468 us; error 68 us; skew 500 ppm
I20260927 13:50:00.395962 27218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:00 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517000, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:00.399537 27218 init.cc:377] Logged in from keytab as oryx/127.20.133.65@KRBTEST.COM (short username oryx)
I20260927 13:50:00.401255 27218 webserver.cc:460] Webserver started at http://127.20.133.65:35185/ using document root <none> and password file <none>
I20260927 13:50:00.402011 27218 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:00.402141 27218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:00.402449 27218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:00.405726 27218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "26bd1ea161b74c5aab11c5a5d20a2e43"
format_stamp: "Formatted at 2026-09-27 13:50:00 on dist-test-slave-q25r"
server_key: "548a2e926683ff55137189c7e3be1e81"
server_key_iv: "199193dc38caf646e7c97efcbc1504ff"
server_key_version: "encryptionkey@0"
I20260927 13:50:00.406672 27218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "26bd1ea161b74c5aab11c5a5d20a2e43"
format_stamp: "Formatted at 2026-09-27 13:50:00 on dist-test-slave-q25r"
server_key: "548a2e926683ff55137189c7e3be1e81"
server_key_iv: "199193dc38caf646e7c97efcbc1504ff"
server_key_version: "encryptionkey@0"
I20260927 13:50:00.413128 27218 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 13:50:00.417347 27233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:00.419730 27218 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:50:00.420130 27218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "26bd1ea161b74c5aab11c5a5d20a2e43"
format_stamp: "Formatted at 2026-09-27 13:50:00 on dist-test-slave-q25r"
server_key: "548a2e926683ff55137189c7e3be1e81"
server_key_iv: "199193dc38caf646e7c97efcbc1504ff"
server_key_version: "encryptionkey@0"
I20260927 13:50:00.420483 27218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:00.440701 27218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:00.446759 27218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:00.447057 27218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:00.448084 27218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:00.449631 27218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:00.449723 27218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:00.449848 27218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:00.449901 27218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:00.484869 27218 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:38349
I20260927 13:50:00.484910 27345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:38349 every 8 connection(s)
I20260927 13:50:00.486192 27218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:50:00.491868 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 27218
I20260927 13:50:00.492057 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:50:00.492761 21013 external_mini_cluster.cc:1428] Setting key 7ea004b84ca9d57f395ba3edc99434ab
Sep 27 13:50:00 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517000, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:50:00.509280 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:00.488850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:45687 (local address 127.20.133.126:33783)
0927 13:50:00.489389 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:00.489394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:00.490271 (+ 877us) server_negotiation.cc:407] Connection header received
0927 13:50:00.492364 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:00.492373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:00.492474 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:00.492589 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:00.494505 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:00.495396 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:00.497864 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:00.498337 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:00.498456 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:00.502795 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:00.502805 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:00.502809 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:00.502858 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:00.506048 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:00.506823 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:00.506828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:00.506831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:00.506970 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:00.507375 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:00.507379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:00.507382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:00.507707 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:00.507849 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:00.508683 (+ 834us) server_negotiation.cc:299] Negotiation successful
0927 13:50:00.508921 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":261,"threads_started":1}
I20260927 13:50:00.510164 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:00.489344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.65:45687)
0927 13:50:00.490100 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:00.490147 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:00.491979 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:00.492912 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:00.492927 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:00.493519 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:00.494282 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:00.494328 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:00.495596 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:00.495601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:00.497637 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:00.497658 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:00.498596 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:00.498602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:00.498902 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:00.499686 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:00.499721 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:00.502483 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:00.506226 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:00.506234 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:00.506250 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:00.506708 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:00.507073 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:00.507077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:00.507079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:00.507284 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:00.507857 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:00.507864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:00.508250 (+ 386us) client_negotiation.cc:769] Sending connection context
0927 13:50:00.508562 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 13:50:00.509002 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":278,"threads_started":1}
I20260927 13:50:00.512008 27346 heartbeater.cc:347] Connected to a master server at 127.20.133.126:33783
I20260927 13:50:00.512384 27346 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:00.513684 27346 heartbeater.cc:511] Master 127.20.133.126:33783 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.20.133.66@KRBTEST.COM; defaulting to no policy
I20260927 13:50:00.516887 27157 ts_manager.cc:194] Registered new tserver with Master: 26bd1ea161b74c5aab11c5a5d20a2e43 (127.20.133.65:38349)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.66@KRBTEST.COM" created.
I20260927 13:50:00.518853 27157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:45687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab.
Entry for principal oryx/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab.
I20260927 13:50:00.543860 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:33783
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab
--principal=oryx/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:00.876914 27352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:00.877255 27352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:00.877393 27352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:00.885404 27352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:00.885550 27352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:00.885703 27352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:50:00.896410 27352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab
--principal=oryx/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:33783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:00.897780 27352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:00.899084 27352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:00.912686 27360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:00.912707 27358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:00.912762 27357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:00.913552 27352 server_base.cc:1034] running on GCE node
I20260927 13:50:00.914171 27352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:00.915174 27352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:00.916416 27352 hybrid_clock.cc:648] HybridClock initialized: now 1790517000916364 us; error 39 us; skew 500 ppm
I20260927 13:50:00.916720 27352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:00 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517000, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:00.919831 27352 init.cc:377] Logged in from keytab as oryx/127.20.133.66@KRBTEST.COM (short username oryx)
I20260927 13:50:00.921434 27352 webserver.cc:460] Webserver started at http://127.20.133.66:43741/ using document root <none> and password file <none>
I20260927 13:50:00.922133 27352 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:00.922233 27352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:00.922479 27352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:00.925546 27352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "be53d36199114b9fbb19b95c28e8ee48"
format_stamp: "Formatted at 2026-09-27 13:50:00 on dist-test-slave-q25r"
server_key: "efda4566e804acdf4d6e687ebbe89394"
server_key_iv: "8f3d63a38b232126cf401c4110a8f60f"
server_key_version: "encryptionkey@0"
I20260927 13:50:00.926363 27352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "be53d36199114b9fbb19b95c28e8ee48"
format_stamp: "Formatted at 2026-09-27 13:50:00 on dist-test-slave-q25r"
server_key: "efda4566e804acdf4d6e687ebbe89394"
server_key_iv: "8f3d63a38b232126cf401c4110a8f60f"
server_key_version: "encryptionkey@0"
I20260927 13:50:00.931551 27352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:50:00.935041 27367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:00.936493 27352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 13:50:00.936655 27352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "be53d36199114b9fbb19b95c28e8ee48"
format_stamp: "Formatted at 2026-09-27 13:50:00 on dist-test-slave-q25r"
server_key: "efda4566e804acdf4d6e687ebbe89394"
server_key_iv: "8f3d63a38b232126cf401c4110a8f60f"
server_key_version: "encryptionkey@0"
I20260927 13:50:00.936860 27352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:00.956099 27352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:00.960784 27352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:00.961048 27352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:00.961946 27352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:00.963251 27352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:00.963344 27352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:00.963434 27352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:00.963478 27352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:00.995420 27352 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:46135
I20260927 13:50:00.995453 27479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:46135 every 8 connection(s)
I20260927 13:50:00.996814 27352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:50:01.006335 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 27352
I20260927 13:50:01.006534 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:50:01.007114 21013 external_mini_cluster.cc:1428] Setting key c5f06f4cc22e86f56744425491c2b9be
Sep 27 13:50:01 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517000, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:50:01.018543 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:00.999462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:36955 (local address 127.20.133.126:33783)
0927 13:50:00.999668 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:00.999672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:01.000892 (+ 1220us) server_negotiation.cc:407] Connection header received
0927 13:50:01.002761 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:01.002767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:01.002857 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:01.002943 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:01.004717 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:01.005442 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:01.007970 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:01.008488 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:01.008605 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:01.012740 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:01.012746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:01.012749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:01.012788 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:01.015296 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:01.016123 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:01.016128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:01.016130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:01.016260 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:01.016745 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:01.016749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:01.016751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:01.017041 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:01.017180 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:01.017940 (+ 760us) server_negotiation.cc:299] Negotiation successful
0927 13:50:01.018189 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 13:50:01.019555 27482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:00.999942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.66:36955)
0927 13:50:01.000731 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:01.000773 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:01.002479 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:01.003206 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:01.003214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:01.003767 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:01.004406 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:01.004427 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:01.005708 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:01.005712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:01.007788 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:01.007805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:01.008733 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:01.008741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:01.008982 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:01.009680 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:01.009721 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:01.012458 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:01.015470 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:01.015482 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:01.015500 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:01.015984 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:01.016406 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:01.016412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:01.016415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:01.016622 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:01.017176 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:01.017184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:01.017562 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 13:50:01.017827 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 13:50:01.018402 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":329,"threads_started":1}
I20260927 13:50:01.021095 27480 heartbeater.cc:347] Connected to a master server at 127.20.133.126:33783
I20260927 13:50:01.021440 27480 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:01.022348 27480 heartbeater.cc:511] Master 127.20.133.126:33783 requested a full tablet report, sending...
I20260927 13:50:01.024170 27157 ts_manager.cc:194] Registered new tserver with Master: be53d36199114b9fbb19b95c28e8ee48 (127.20.133.66:46135)
I20260927 13:50:01.025391 27157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:36955
WARNING: no policy specified for oryx/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab.
Entry for principal oryx/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab.
I20260927 13:50:01.056717 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:33783
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab
--principal=oryx/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:01.382184 27485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:01.382473 27485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:01.382576 27485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:01.390332 27485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:01.390462 27485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:01.390594 27485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:50:01.401147 27485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab
--principal=oryx/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:33783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:01.402437 27485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:01.403791 27485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:01.418140 27493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:01.418241 27490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:01.418221 27491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:01.418831 27485 server_base.cc:1034] running on GCE node
I20260927 13:50:01.419564 27485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:01.420642 27485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:01.421877 27485 hybrid_clock.cc:648] HybridClock initialized: now 1790517001421811 us; error 59 us; skew 500 ppm
I20260927 13:50:01.422206 27485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:01 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517001, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:01.425398 27485 init.cc:377] Logged in from keytab as oryx/127.20.133.67@KRBTEST.COM (short username oryx)
I20260927 13:50:01.426928 27485 webserver.cc:460] Webserver started at http://127.20.133.67:36235/ using document root <none> and password file <none>
I20260927 13:50:01.427619 27485 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:01.427737 27485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:01.428046 27485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:01.431113 27485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "9e26d452393e4256b18196c585186f24"
format_stamp: "Formatted at 2026-09-27 13:50:01 on dist-test-slave-q25r"
server_key: "d8772a152c76d941820dee5614f220ab"
server_key_iv: "7d77fc6f1fdc212ca9008813ad73f030"
server_key_version: "encryptionkey@0"
I20260927 13:50:01.431865 27485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "9e26d452393e4256b18196c585186f24"
format_stamp: "Formatted at 2026-09-27 13:50:01 on dist-test-slave-q25r"
server_key: "d8772a152c76d941820dee5614f220ab"
server_key_iv: "7d77fc6f1fdc212ca9008813ad73f030"
server_key_version: "encryptionkey@0"
I20260927 13:50:01.437288 27485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:50:01.440868 27500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:01.442391 27485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:50:01.442557 27485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "9e26d452393e4256b18196c585186f24"
format_stamp: "Formatted at 2026-09-27 13:50:01 on dist-test-slave-q25r"
server_key: "d8772a152c76d941820dee5614f220ab"
server_key_iv: "7d77fc6f1fdc212ca9008813ad73f030"
server_key_version: "encryptionkey@0"
I20260927 13:50:01.442761 27485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:01.456132 27485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:01.460039 27485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:01.460316 27485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:01.461339 27485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:01.462729 27485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:01.462807 27485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:01.462929 27485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:01.462985 27485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:01.493173 27485 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:44205
I20260927 13:50:01.493209 27612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:44205 every 8 connection(s)
I20260927 13:50:01.494388 27485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:50:01.499394 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 27485
I20260927 13:50:01.499567 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:50:01.500128 21013 external_mini_cluster.cc:1428] Setting key f25d003f065cf36ba827c47c3ed80a81
Sep 27 13:50:01 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517001, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:50:01.514439 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:01.496885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:40703 (local address 127.20.133.126:33783)
0927 13:50:01.497054 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:01.497059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:01.498184 (+ 1125us) server_negotiation.cc:407] Connection header received
0927 13:50:01.500445 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:01.500451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:01.500524 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:01.500610 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:01.502553 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:01.503408 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:01.505037 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:01.505461 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:01.505573 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:01.508978 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:01.508983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:01.508986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:01.509019 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:01.511376 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:01.512087 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:01.512091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:01.512093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:01.512220 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:01.512698 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:01.512701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:01.512703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:01.512978 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:01.513125 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:01.513821 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 13:50:01.514057 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:50:01.515483 27615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:01.497345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.67:40703)
0927 13:50:01.498032 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:01.498072 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:01.500118 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:01.500864 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:01.500874 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:01.501541 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:01.502377 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:01.502405 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:01.503561 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:01.503567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:01.504899 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:01.504911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:01.505685 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:01.505692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:01.505940 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:01.506608 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:01.506632 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:01.508772 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:01.511519 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:01.511529 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:01.511547 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:01.511964 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:01.512327 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:01.512357 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:01.512360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:01.512582 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:01.513105 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:01.513113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:01.513463 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 13:50:01.513694 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 13:50:01.514246 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":267,"threads_started":1}
I20260927 13:50:01.516956 27613 heartbeater.cc:347] Connected to a master server at 127.20.133.126:33783
I20260927 13:50:01.517279 27613 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:01.518074 27613 heartbeater.cc:511] Master 127.20.133.126:33783 requested a full tablet report, sending...
I20260927 13:50:01.519650 27157 ts_manager.cc:194] Registered new tserver with Master: 9e26d452393e4256b18196c585186f24 (127.20.133.67:44205)
I20260927 13:50:01.520555 27157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:40703
I20260927 13:50:01.522483 27346 heartbeater.cc:503] Master 127.20.133.126:33783 was elected leader, sending a full tablet report...
I20260927 13:50:01.528391 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.133.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.125.keytab.
Entry for principal oryx/127.20.133.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.125.keytab.
I20260927 13:50:01.576254 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.125:33637
--webserver_interface=127.20.133.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.125.keytab
--principal=oryx/127.20.133.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:33783,127.20.133.125:33637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:01.906072 27618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:01.906399 27618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:01.906497 27618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:01.914254 27618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:50:01.914402 27618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:01.914470 27618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:01.914534 27618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:50:01.914584 27618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:50:01.925957 27618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:33783,127.20.133.125:33637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.125.keytab
--principal=oryx/127.20.133.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.125:33637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.133.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
W20260927 13:50:01.926317 27618 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.133.126:33783,127.20.133.125:33637'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 13:50:01.927596 27618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:01.929157 27618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:01.942344 27623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:01.942569 27626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:01.942400 27624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:01.943667 27618 server_base.cc:1034] running on GCE node
I20260927 13:50:01.944927 27618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:01.946480 27618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:01.947896 27618 hybrid_clock.cc:648] HybridClock initialized: now 1790517001947760 us; error 67 us; skew 500 ppm
I20260927 13:50:01.948541 27618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:01 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517001, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:01.952602 27618 init.cc:377] Logged in from keytab as oryx/127.20.133.125@KRBTEST.COM (short username oryx)
I20260927 13:50:01.954355 27618 webserver.cc:460] Webserver started at http://127.20.133.125:43631/ using document root <none> and password file <none>
I20260927 13:50:01.955143 27618 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:01.955255 27618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:01.955904 27618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:01.961135 27618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data/instance:
uuid: "858b03dc507a47ea9997ce955053284a"
format_stamp: "Formatted at 2026-09-27 13:50:01 on dist-test-slave-q25r"
server_key: "a4095a661dd825a8a1cb59a438137d4f"
server_key_iv: "471a340d95fe03a800de82d6156c7339"
server_key_version: "encryptionkey@0"
I20260927 13:50:01.962251 27618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/wal/instance:
uuid: "858b03dc507a47ea9997ce955053284a"
format_stamp: "Formatted at 2026-09-27 13:50:01 on dist-test-slave-q25r"
server_key: "a4095a661dd825a8a1cb59a438137d4f"
server_key_iv: "471a340d95fe03a800de82d6156c7339"
server_key_version: "encryptionkey@0"
I20260927 13:50:01.968864 27618 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 13:50:01.973042 27633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:01.974865 27618 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:50:01.975057 27618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/wal
uuid: "858b03dc507a47ea9997ce955053284a"
format_stamp: "Formatted at 2026-09-27 13:50:01 on dist-test-slave-q25r"
server_key: "a4095a661dd825a8a1cb59a438137d4f"
server_key_iv: "471a340d95fe03a800de82d6156c7339"
server_key_version: "encryptionkey@0"
I20260927 13:50:01.975297 27618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:02.006987 27618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:02.011978 27618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:02.012253 27618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:02.028568 27480 heartbeater.cc:503] Master 127.20.133.126:33783 was elected leader, sending a full tablet report...
I20260927 13:50:02.031857 27618 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.125:33637
I20260927 13:50:02.031876 27684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.125:33637 every 8 connection(s)
I20260927 13:50:02.033792 27618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
I20260927 13:50:02.037681 27685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:02.037994 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 27618
I20260927 13:50:02.049399 27685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.133.125@KRBTEST.COM
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517001, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.125@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:50:02.065057 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.040227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.20.133.125:33637)
0927 13:50:02.041131 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.041168 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.041225 (+ 57us) server_negotiation.cc:407] Connection header received
0927 13:50:02.043575 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.043609 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.044140 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.044650 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.047032 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.048483 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.050352 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.051146 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.051285 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.056711 (+ 5426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.056721 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.056734 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.056776 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.060091 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.061032 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.061040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.061047 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.061226 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.061732 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.061736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.061738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.062357 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.062658 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.063077 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.063546 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":387,"threads_started":1}
I20260927 13:50:02.069815 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.052750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.20.133.126:33783)
0927 13:50:02.054050 (+ 1300us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.054055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.054079 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:50:02.054129 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.054133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.054188 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.054280 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.055460 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.056612 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.058310 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.058740 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.058853 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.062474 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.062481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.062486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.062532 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.067003 (+ 4471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.067887 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.067892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.067895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.068066 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.068482 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.068487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.068490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.068829 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.068987 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.069362 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.069550 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1204,"thread_start_us":1092,"threads_started":1}
I20260927 13:50:02.069837 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.052716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:43922)
0927 13:50:02.053462 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.053512 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.053688 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.054390 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.054398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.055040 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.055300 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.055310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.056758 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.056764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.058143 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.058164 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.058930 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.058937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.059116 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.059735 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.059755 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.062243 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.067200 (+ 4957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.067210 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.067220 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.067740 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.068199 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.068203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.068205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.068387 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.068941 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.068948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.069121 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.069277 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.069543 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":349,"threads_started":1}
I20260927 13:50:02.071468 27685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517001, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.125@KRBTEST.COM for oryx/127.20.133.125@KRBTEST.COM
I20260927 13:50:02.083787 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.072251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:33637 (local address 127.0.0.1:54634)
0927 13:50:02.072466 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.072484 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.072596 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.072869 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.072874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.073305 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.073487 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.073497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.074435 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.074440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.075300 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.075311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.075979 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.075984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.076163 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.076726 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.076742 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.078783 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.081582 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.081587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.081592 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.081953 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.082272 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.082277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.082281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.082402 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.082970 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.082974 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.083127 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.083261 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.083518 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 13:50:02.083983 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.072219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.20.133.125:33637)
0927 13:50:02.072459 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.072465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.072542 (+ 77us) server_negotiation.cc:407] Connection header received
0927 13:50:02.072705 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.072708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.072761 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.072841 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.073610 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.074298 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.075428 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.075835 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.075943 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.078971 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.078975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.078978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.079007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.081447 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.082060 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.082066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.082068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.082175 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.082492 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.082497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.082500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.082804 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.082952 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.083359 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.083672 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":68}
I20260927 13:50:02.088836 27685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: Bootstrap starting.
I20260927 13:50:02.091929 27685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:02.092999 27685 log.cc:824] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:02.095808 27685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: No bootstrap required, opened a new log
I20260927 13:50:02.103942 27685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } }
I20260927 13:50:02.104321 27685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:02.104467 27685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858b03dc507a47ea9997ce955053284a, State: Initialized, Role: FOLLOWER
I20260927 13:50:02.105093 27685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } }
I20260927 13:50:02.106000 27692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } } }
I20260927 13:50:02.106415 27692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:50:02.107034 27685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: configured and running, proceeding with master startup.
W20260927 13:50:02.116714 27703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:50:02.116879 27703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:50:02.120091 27618 master_runner.cc:428] Detected that this master 858b03dc507a47ea9997ce955053284a is joining an existing cluster
I20260927 13:50:02.120218 27618 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.133.125:33637
I20260927 13:50:02.144282 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.126799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.20.133.126:33783)
0927 13:50:02.127048 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.127054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.129402 (+ 2348us) server_negotiation.cc:407] Connection header received
0927 13:50:02.129701 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.129708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.129799 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.129855 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.131055 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.132090 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.134340 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.134905 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.134992 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.137536 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.137542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.137547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.137589 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.141067 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.141910 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.141915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.141918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.142075 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.142522 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.142527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.142530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.142908 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.143031 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.143662 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.143925 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260927 13:50:02.145552 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.126326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:43932)
0927 13:50:02.129256 (+ 2930us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.129278 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.129532 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.130032 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.130035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.130580 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.130896 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.130908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.132267 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.132272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.134176 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.134192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.135315 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.135321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.135535 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.136181 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.136204 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.137313 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.141293 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.141298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.141301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.141782 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.142230 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.142234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.142237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.142401 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.143108 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.143112 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.143351 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.144589 (+ 1238us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.144974 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2762,"thread_start_us":18447,"threads_started":1}
WARNING: no policy specified for oryx/127.20.133.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.124@KRBTEST.COM" created.
I20260927 13:50:02.161209 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.126433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.20.133.125:33637)
0927 13:50:02.126617 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.126624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.144823 (+ 18199us) server_negotiation.cc:407] Connection header received
0927 13:50:02.145036 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.145040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.145105 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.145175 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.147874 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.149094 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.151124 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.151817 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.151933 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.154196 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.154202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.154207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.154246 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.157870 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.158895 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.158900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.158904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.159066 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.159490 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.159495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.159499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.159875 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.160013 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.160576 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.160816 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:50:02.162273 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.126711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:33637 (local address 127.0.0.1:54644)
0927 13:50:02.144724 (+ 18013us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.144747 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.144915 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.145356 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.145360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.147219 (+ 1859us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.147671 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.147690 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.149317 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.149323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.150944 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.150960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.152036 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.152043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.152236 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.152822 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.152847 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.153985 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.158023 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.158028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.158032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.158545 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.159200 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.159205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.159208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.159383 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.160095 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.160099 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.160239 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.161516 (+ 1277us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.161823 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17864,"thread_start_us":2606,"threads_started":1}
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516999, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.126@KRBTEST.COM for oryx/127.20.133.125@KRBTEST.COM
I20260927 13:50:02.179455 27692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:50:02.179809 27692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } }
I20260927 13:50:02.181253 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.153891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:33637 (local address 127.0.0.1:54652)
0927 13:50:02.163327 (+ 9436us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.163379 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.163636 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.164453 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.164461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.165438 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.165792 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.165813 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.167114 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.167120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.168178 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.168194 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.169329 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.169336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.169579 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.170364 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.170392 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.173457 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.177815 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.177836 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.177845 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.178527 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.178969 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.178975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.178978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.179174 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.180110 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.180119 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.180376 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.180605 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.180887 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9107,"thread_start_us":10151,"threads_started":1}
I20260927 13:50:02.181493 27692 leader_election.cc:288] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783)
I20260927 13:50:02.182406 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.153943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.20.133.125:33637)
0927 13:50:02.161481 (+ 7538us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.161486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.163464 (+ 1978us) server_negotiation.cc:407] Connection header received
0927 13:50:02.163914 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.163920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.164038 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.164113 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.165989 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.166955 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.168381 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.169000 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.169247 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.173725 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.173733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.173738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.173784 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.177599 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.178693 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.178700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.178703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.178861 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.179326 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.179332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.179335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.179828 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.180028 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.181739 (+ 1711us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.181989 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7439,"thread_start_us":12021,"threads_started":1}
I20260927 13:50:02.182391 27167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "858b03dc507a47ea9997ce955053284a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25826f702bd249f280d27611d1ce1ad2" is_pre_election: true
I20260927 13:50:02.183616 27634 leader_election.cc:302] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 858b03dc507a47ea9997ce955053284a; no voters: 25826f702bd249f280d27611d1ce1ad2
I20260927 13:50:02.184036 27692 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 13:50:02.185082 27157 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.133.125:33637: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "858b03dc507a47ea9997ce955053284a"
member_type: NON_VOTER
last_known_addr {
host: "127.20.133.125"
port: 33637
}
attrs {
promote: true
}
}
dest_uuid: "25826f702bd249f280d27611d1ce1ad2"
cas_config_opid_index: -1
I20260927 13:50:02.186198 27157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.124.keytab.
Entry for principal oryx/127.20.133.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.124.keytab.
I20260927 13:50:02.191447 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.124:38229
--webserver_interface=127.20.133.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.124.keytab
--principal=oryx/127.20.133.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:33783,127.20.133.125:33637,127.20.133.124:38229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:50:02.198436 27716 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 858b03dc507a47ea9997ce955053284a (127.20.133.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } }
I20260927 13:50:02.200122 27660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:02.203799 27660 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 FOLLOWER]: Refusing update from remote peer 25826f702bd249f280d27611d1ce1ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 13:50:02.214438 27720 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:02.216645 27716 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.133.125:33637
I20260927 13:50:02.217355 27717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } }
I20260927 13:50:02.217737 27717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:02.218580 27618 master.cc:537] Master@127.20.133.125:33637 shutting down...
I20260927 13:50:02.260396 27660 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 858b03dc507a47ea9997ce955053284a (127.20.133.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } }
I20260927 13:50:02.266183 27692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25826f702bd249f280d27611d1ce1ad2. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } }
I20260927 13:50:02.266510 27692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: This master's current role is: LEARNER
I20260927 13:50:02.266235 27724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } }
I20260927 13:50:02.266852 27724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: This master's current role is: LEARNER
I20260927 13:50:02.266045 27727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } }
I20260927 13:50:02.267233 27727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: This master's current role is: LEARNER
I20260927 13:50:02.269459 27618 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 LEARNER]: Raft consensus shutting down.
I20260927 13:50:02.269975 27618 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 LEARNER]: Raft consensus is shut down!
I20260927 13:50:02.270147 27618 tablet_replica.cc:330] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: stopping tablet replica
I20260927 13:50:02.288887 27618 master.cc:552] Master@127.20.133.125:33637 shutdown complete.
I20260927 13:50:02.289006 27618 master_runner.cc:305] Clearing existing system tablet
I20260927 13:50:02.289459 27618 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:02.289587 27618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:02.292739 27618 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 13:50:02.294605 27729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:02.295070 27618 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:02.295384 27618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:50:02.295492 27618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/wal
uuid: "858b03dc507a47ea9997ce955053284a"
format_stamp: "Formatted at 2026-09-27 13:50:01 on dist-test-slave-q25r"
server_key: "a4095a661dd825a8a1cb59a438137d4f"
server_key_iv: "471a340d95fe03a800de82d6156c7339"
server_key_version: "encryptionkey@0"
I20260927 13:50:02.300747 27618 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:02.308329 27618 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 13:50:02.308581 27618 log.cc:1196] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 13:50:02.309094 27618 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: Deleting consensus metadata
I20260927 13:50:02.309358 27618 master_runner.cc:315] Copying system tablet from 127.20.133.126:33783
I20260927 13:50:02.318041 27618 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.133.126:33783
I20260927 13:50:02.334573 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.319048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.20.133.126:33783)
0927 13:50:02.319255 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.319261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.319827 (+ 566us) server_negotiation.cc:407] Connection header received
0927 13:50:02.320074 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.320081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.320163 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.320267 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.321480 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.322617 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.324509 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.325214 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.325346 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.327800 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.327808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.327813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.327856 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.331302 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.332222 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.332229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.332233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.332440 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.332875 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.332880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.332883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.333276 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.333429 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.333967 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.334201 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 13:50:02.334614 27735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.318913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:43934)
0927 13:50:02.319713 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.319740 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.319901 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.320365 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.320370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.320925 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.321248 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.321263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.322798 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.322804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.324321 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.324371 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.325418 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.325425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.325613 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.326394 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.326420 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.327563 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.331477 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.331482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.331486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.332069 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.332563 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.332568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.332572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.332741 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.333407 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.333412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.333599 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.333778 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.334133 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":508,"threads_started":1}
I20260927 13:50:02.335947 27177 tablet_copy_service.cc:140] P 25826f702bd249f280d27611d1ce1ad2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 858b03dc507a47ea9997ce955053284a ({username='oryx', principal='oryx/127.20.133.125@KRBTEST.COM'} at 127.0.0.1:43934)
I20260927 13:50:02.336228 27177 tablet_copy_service.cc:161] P 25826f702bd249f280d27611d1ce1ad2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 858b03dc507a47ea9997ce955053284a at {username='oryx', principal='oryx/127.20.133.125@KRBTEST.COM'} at 127.0.0.1:43934: session id = 858b03dc507a47ea9997ce955053284a-00000000000000000000000000000000
I20260927 13:50:02.340786 27177 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 13:50:02.343286 27618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 13:50:02.350364 27618 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: tablet copy: Starting download of 0 data blocks...
I20260927 13:50:02.350714 27618 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: tablet copy: Starting download of 1 WAL segments...
I20260927 13:50:02.353648 27618 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 13:50:02.358466 27177 tablet_copy_service.cc:342] P 25826f702bd249f280d27611d1ce1ad2: Request end of tablet copy session 858b03dc507a47ea9997ce955053284a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.133.125@KRBTEST.COM'} at 127.0.0.1:43934
I20260927 13:50:02.358698 27177 tablet_copy_service.cc:433] P 25826f702bd249f280d27611d1ce1ad2: ending tablet copy session 858b03dc507a47ea9997ce955053284a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 858b03dc507a47ea9997ce955053284a
I20260927 13:50:02.370390 27618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:02.373162 27736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:02.373633 27737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:02.384645 27739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:02.389916 27618 server_base.cc:1034] running on GCE node
I20260927 13:50:02.390333 27618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:02.391058 27618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:02.392311 27618 hybrid_clock.cc:648] HybridClock initialized: now 1790517002392243 us; error 42 us; skew 500 ppm
I20260927 13:50:02.392571 27618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517002, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:02.395140 27618 init.cc:377] Logged in from keytab as oryx/127.20.133.125@KRBTEST.COM (short username oryx)
I20260927 13:50:02.397429 27618 webserver.cc:460] Webserver started at http://127.20.133.125:35349/ using document root <none> and password file <none>
I20260927 13:50:02.397857 27618 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:02.397945 27618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:02.401051 27618 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 13:50:02.403478 27746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:02.403980 27618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 13:50:02.404301 27618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/wal
uuid: "858b03dc507a47ea9997ce955053284a"
format_stamp: "Formatted at 2026-09-27 13:50:01 on dist-test-slave-q25r"
server_key: "a4095a661dd825a8a1cb59a438137d4f"
server_key_iv: "471a340d95fe03a800de82d6156c7339"
server_key_version: "encryptionkey@0"
I20260927 13:50:02.404647 27618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:02.424636 27618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:02.429034 27618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:02.455539 27618 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.125:33637
I20260927 13:50:02.455585 27797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.125:33637 every 8 connection(s)
I20260927 13:50:02.457227 27618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
I20260927 13:50:02.462658 27798 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:50:02.466627 27798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: Bootstrap starting.
I20260927 13:50:02.482322 27798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:50:02.482883 27798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: Bootstrap complete.
I20260927 13:50:02.484021 27798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } }
I20260927 13:50:02.484309 27798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 858b03dc507a47ea9997ce955053284a, State: Initialized, Role: LEARNER
I20260927 13:50:02.484578 27798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } }
I20260927 13:50:02.489233 27798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:50:02.489107 27801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } }
I20260927 13:50:02.489759 27801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: This master's current role is: LEARNER
I20260927 13:50:02.508482 27812 catalog_manager.cc:1259] Loaded cluster ID: a47906cfaa1d4b80bc3aac7116ef0a20
I20260927 13:50:02.508659 27812 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: loading cluster ID for follower catalog manager: success
I20260927 13:50:02.512455 27812 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: acquiring CA information for follower catalog manager: success
I20260927 13:50:02.515363 27812 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:50:02.523561 27613 heartbeater.cc:503] Master 127.20.133.126:33783 was elected leader, sending a full tablet report...
W20260927 13:50:02.614454 27718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:02.614815 27718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:02.614965 27718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:02.622648 27718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:50:02.622787 27718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:02.622836 27718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:02.622916 27718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:50:02.622973 27718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:50:02.633478 27718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:33783,127.20.133.125:33637,127.20.133.124:38229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.124.keytab
--principal=oryx/127.20.133.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.124:38229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.133.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:02.634939 27718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:02.636260 27718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:02.648780 27817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:02.648722 27818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:02.648751 27820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:02.649536 27718 server_base.cc:1034] running on GCE node
I20260927 13:50:02.650441 27718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:02.652019 27718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:02.653263 27718 hybrid_clock.cc:648] HybridClock initialized: now 1790517002653201 us; error 62 us; skew 500 ppm
I20260927 13:50:02.653700 27718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517002, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:02.657125 27718 init.cc:377] Logged in from keytab as oryx/127.20.133.124@KRBTEST.COM (short username oryx)
I20260927 13:50:02.658711 27718 webserver.cc:460] Webserver started at http://127.20.133.124:41749/ using document root <none> and password file <none>
I20260927 13:50:02.659416 27718 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:02.659539 27718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:02.659833 27718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:02.662937 27718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data/instance:
uuid: "cd7597d7a00a4223aabb7168756d8ebd"
format_stamp: "Formatted at 2026-09-27 13:50:02 on dist-test-slave-q25r"
server_key: "ce65cac7f22ff5575b117efd2213fdea"
server_key_iv: "df6d7ca57b29a87c551e77123f93c966"
server_key_version: "encryptionkey@0"
I20260927 13:50:02.663758 27718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/wal/instance:
uuid: "cd7597d7a00a4223aabb7168756d8ebd"
format_stamp: "Formatted at 2026-09-27 13:50:02 on dist-test-slave-q25r"
server_key: "ce65cac7f22ff5575b117efd2213fdea"
server_key_iv: "df6d7ca57b29a87c551e77123f93c966"
server_key_version: "encryptionkey@0"
I20260927 13:50:02.669413 27718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 13:50:02.673069 27827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:02.674474 27718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:50:02.674649 27718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/wal
uuid: "cd7597d7a00a4223aabb7168756d8ebd"
format_stamp: "Formatted at 2026-09-27 13:50:02 on dist-test-slave-q25r"
server_key: "ce65cac7f22ff5575b117efd2213fdea"
server_key_iv: "df6d7ca57b29a87c551e77123f93c966"
server_key_version: "encryptionkey@0"
I20260927 13:50:02.674851 27718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:02.690271 27718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:02.695000 27718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:02.695276 27718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:02.712826 27718 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.124:38229
I20260927 13:50:02.712867 27878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.124:38229 every 8 connection(s)
I20260927 13:50:02.714262 27718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
I20260927 13:50:02.718443 27879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:02.722543 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 27718
I20260927 13:50:02.725659 27879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517002, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.124@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.133.124@KRBTEST.COM
I20260927 13:50:02.746366 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.729654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.20.133.126:33783)
0927 13:50:02.729859 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.729865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.730334 (+ 469us) server_negotiation.cc:407] Connection header received
0927 13:50:02.730517 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.730523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.730598 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.730679 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.731902 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.733009 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.734985 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.735513 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.735652 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.739239 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.739247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.739252 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.739312 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.742370 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.743394 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.743401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.743404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.743583 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.744124 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.744128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.744130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.744605 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.744745 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.745731 (+ 986us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.746000 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 13:50:02.747659 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.729440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:43936)
0927 13:50:02.730202 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.730250 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.730386 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.730754 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.730761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.731421 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.731724 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.731738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.733192 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.733196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.734743 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.734756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.735737 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.735743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.735891 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.736481 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.736505 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.738999 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.742538 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.742551 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.742566 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.743241 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.743759 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.743763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.743765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.743984 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.744725 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.744734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.745298 (+ 564us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.745619 (+ 321us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.746166 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":36,"thread_start_us":359,"threads_started":1}
I20260927 13:50:02.747680 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.724794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.20.133.124:38229)
0927 13:50:02.725697 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.725711 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.725760 (+ 49us) server_negotiation.cc:407] Connection header received
0927 13:50:02.727705 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.727729 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.728178 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.728572 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.730246 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.731743 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.733493 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.734552 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.734767 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.739726 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.739734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.739748 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.739785 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.743075 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.744017 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.744026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.744029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.744192 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.744689 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.744694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.744697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.745294 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.745619 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.745960 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.746344 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":622,"thread_start_us":391,"threads_started":1}
I20260927 13:50:02.749753 27879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517002, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.124@KRBTEST.COM for oryx/127.20.133.125@KRBTEST.COM
I20260927 13:50:02.763141 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.750558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:33637 (local address 127.0.0.1:54654)
0927 13:50:02.750773 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.750794 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.750904 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.751855 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.751858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.752228 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.752425 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.752436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.753716 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.753719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.754640 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.754651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.755292 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.755296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.755424 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.755965 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.755982 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.758046 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.760908 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.760912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.760915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.761301 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.761695 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.761698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.761706 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.761834 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.762328 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.762332 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.762489 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.762642 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.762889 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 13:50:02.763232 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.750993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.20.133.125:33637)
0927 13:50:02.751526 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.751531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.751547 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:50:02.751619 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.751623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.751694 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.751787 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.752589 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.753591 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.754778 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.755185 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.755283 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.758252 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.758257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.758260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.758294 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.760771 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.761427 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.761431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.761434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.761605 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.761915 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.761918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.761920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.762197 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.762319 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.762779 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.762964 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":275,"threads_started":1}
I20260927 13:50:02.764313 27879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517002, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.124@KRBTEST.COM for oryx/127.20.133.124@KRBTEST.COM
I20260927 13:50:02.777021 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.765113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:38229 (local address 127.0.0.1:52350)
0927 13:50:02.765304 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.765323 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.765425 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.765720 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.765724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.766078 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.766218 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.766226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.767198 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.767203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.768110 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.768123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.768966 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.768971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.769141 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.769724 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.769741 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.771968 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.774847 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.774851 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.774854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.775223 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.775561 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.775564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.775566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.775721 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.776184 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.776187 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.776307 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.776484 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.776717 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 13:50:02.777086 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.765279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.20.133.124:38229)
0927 13:50:02.765456 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.765460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.765476 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:50:02.765515 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.765519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.765573 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.765695 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.766334 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.767041 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.768260 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.768769 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.768980 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.772182 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.772188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.772191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.772224 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.774687 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.775334 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.775339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.775341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.775456 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.775808 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.775811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.775813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.776087 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.776203 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.776606 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.776839 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":78}
I20260927 13:50:02.781793 27879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: Bootstrap starting.
I20260927 13:50:02.785072 27879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:02.786121 27879 log.cc:824] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:02.788805 27879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: No bootstrap required, opened a new log
I20260927 13:50:02.796422 27879 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } }
I20260927 13:50:02.796810 27879 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:02.796926 27879 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7597d7a00a4223aabb7168756d8ebd, State: Initialized, Role: FOLLOWER
I20260927 13:50:02.797525 27879 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } }
I20260927 13:50:02.798414 27886 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } } }
I20260927 13:50:02.798730 27886 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:50:02.799413 27879 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: configured and running, proceeding with master startup.
W20260927 13:50:02.808074 27897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:50:02.808247 27897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:50:02.812394 27718 master_runner.cc:428] Detected that this master cd7597d7a00a4223aabb7168756d8ebd is joining an existing cluster
I20260927 13:50:02.812510 27718 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.133.124:38229
I20260927 13:50:02.836030 27905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.816805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:43946)
0927 13:50:02.820840 (+ 4035us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.820859 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.821045 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.821435 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.821440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.822003 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.822309 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.822319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.823714 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.823719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.825579 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.825590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.826801 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.826807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.827006 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.827783 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.827807 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.829063 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.832700 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.832705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.832709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.833358 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.833816 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.833821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.833824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.834027 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.834767 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.834772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.835014 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.835171 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.835531 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3876,"mutex_wait_us":3,"thread_start_us":5779,"threads_started":1}
I20260927 13:50:02.838853 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.816694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.20.133.124:38229)
0927 13:50:02.816940 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.816945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.823025 (+ 6080us) server_negotiation.cc:407] Connection header received
0927 13:50:02.823157 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.823166 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.823251 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.823393 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.824910 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.826051 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.827967 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.828712 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.828818 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.831632 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.831638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.831643 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.831677 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.835294 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.836077 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.836083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.836086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.836272 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.837207 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.837211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.837215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.837576 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.837698 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.838216 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.838448 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":84}
I20260927 13:50:02.840083 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.816852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:38229 (local address 127.0.0.1:52364)
0927 13:50:02.822907 (+ 6055us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.822927 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.823047 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.823560 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.823564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.824300 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.824750 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.824773 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.826263 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.826267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.827822 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.827837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.829128 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.829135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.829338 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.830217 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.830242 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.831442 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.835445 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.835449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.835453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.835949 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.836923 (+ 974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.836927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.836931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.837097 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.837793 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.837798 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.837959 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.839270 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.839578 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5894,"mutex_wait_us":1,"thread_start_us":3413,"threads_started":1}
I20260927 13:50:02.845613 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.817609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.20.133.126:33783)
0927 13:50:02.817974 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.817979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.820985 (+ 3006us) server_negotiation.cc:407] Connection header received
0927 13:50:02.821177 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.821183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.821275 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.821373 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.822468 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.823552 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.825732 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.826392 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.826509 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.829297 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.829305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.829311 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.829353 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.832543 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.833509 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.833515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.833518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.833678 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.834166 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.834171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.834175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.834552 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.834698 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.844917 (+ 10219us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.845244 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260927 13:50:02.845588 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.817401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.20.133.125:33637)
0927 13:50:02.817830 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.817836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.823569 (+ 5733us) server_negotiation.cc:407] Connection header received
0927 13:50:02.823835 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.823841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.823942 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.824015 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.825591 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.826644 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.828761 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.829377 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.829471 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.832184 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.832191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.832197 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.832240 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.835545 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.843052 (+ 7507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.843060 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.843063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.843244 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.843736 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.843742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.843746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.844138 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.844280 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.844983 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.845248 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260927 13:50:02.847325 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.816916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:33637 (local address 127.0.0.1:54664)
0927 13:50:02.823463 (+ 6547us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.823480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.823671 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.824216 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.824221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.825056 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.825399 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.825410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.826847 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.826851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.828573 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.828590 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.829763 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.829771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.829934 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.830527 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.830556 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.831953 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.841722 (+ 9769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.841727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.841731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.842209 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.843418 (+ 1209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.843426 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.843430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.843595 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.844415 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.844421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.844658 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.846382 (+ 1724us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.846830 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6422,"mutex_wait_us":70,"spinlock_wait_cycles":23808,"thread_start_us":4228,"threads_started":1}
I20260927 13:50:02.853669 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 27218
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516999, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.126@KRBTEST.COM for oryx/127.20.133.124@KRBTEST.COM
I20260927 13:50:02.867596 27906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } }
I20260927 13:50:02.868037 27906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:02.873385 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.851730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.20.133.124:38229)
0927 13:50:02.851891 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.851896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.855331 (+ 3435us) server_negotiation.cc:407] Connection header received
0927 13:50:02.855516 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.855522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.855644 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.855741 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.857086 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.858220 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.860301 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.861038 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.861141 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.866036 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.866044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.866049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.866098 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.870000 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.870984 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.870990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.870994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.871168 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.871652 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.871657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.871661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.872031 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.872170 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.872781 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.873027 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:50:02.874630 27908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.851651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:38229 (local address 127.0.0.1:52374)
0927 13:50:02.855221 (+ 3570us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.855240 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.855366 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.855990 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.855996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.856638 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.856907 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.856921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.858457 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.858463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.860092 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.860109 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.861450 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.861457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.861679 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.862499 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.862546 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.865784 (+ 3238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.870182 (+ 4398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.870188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.870192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.870825 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.871328 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.871333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.871337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.871507 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.872258 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.872263 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.872473 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.873876 (+ 1403us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.874188 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3457,"thread_start_us":3279,"threads_started":1}
I20260927 13:50:02.877334 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.857064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:33637 (local address 127.0.0.1:54672)
0927 13:50:02.866674 (+ 9610us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.866700 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.866823 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.867390 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.867396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.867421 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:02.868443 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.868457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.869882 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.869888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.872554 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.872569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.876446 (+ 3877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.876453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.876665 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.876744 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.876842 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.877002 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9501,"thread_start_us":9839,"threads_started":1}
I20260927 13:50:02.877357 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.857427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.20.133.125:33637)
0927 13:50:02.857603 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.857608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.866809 (+ 9201us) server_negotiation.cc:407] Connection header received
0927 13:50:02.866940 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.866946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.867046 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.867169 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:02.868621 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.869686 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.874548 (+ 4862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.875801 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.875919 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.876060 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.876924 (+ 864us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.877091 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:50:02.877540 27156 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.133.124:38229: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd"
member_type: NON_VOTER
last_known_addr {
host: "127.20.133.124"
port: 38229
}
attrs {
promote: true
}
}
dest_uuid: "25826f702bd249f280d27611d1ce1ad2"
cas_config_opid_index: 5
I20260927 13:50:02.878342 27156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } }
I20260927 13:50:02.878759 27801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25826f702bd249f280d27611d1ce1ad2. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } }
I20260927 13:50:02.879005 27801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: This master's current role is: LEARNER
Sep 27 13:50:02 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.133.65@KRBTEST.COM
I20260927 13:50:02.881532 27854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:02.885718 27854 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 FOLLOWER]: Refusing update from remote peer 25826f702bd249f280d27611d1ce1ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 13:50:02.888474 27907 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cd7597d7a00a4223aabb7168756d8ebd (127.20.133.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:02.889557 27773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 LEARNER]: Refusing update from remote peer 25826f702bd249f280d27611d1ce1ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 13:50:02.890331 27912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:02.895157 27907 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.133.124:38229
I20260927 13:50:02.895820 27718 master.cc:537] Master@127.20.133.124:38229 shutting down...
I20260927 13:50:02.896555 27907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:02.896971 27907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:02.898097 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.855261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.20.133.65:38349)
0927 13:50:02.868246 (+ 12985us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.868256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.868289 (+ 33us) server_negotiation.cc:407] Connection header received
0927 13:50:02.868380 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.868389 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.868679 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.870614 (+ 1935us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.870900 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.872436 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.874560 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.875468 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.875607 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.883046 (+ 7439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.883069 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.883091 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.883146 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.891124 (+ 7978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.892280 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.892293 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.892297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.892497 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.893044 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.893049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.893053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.893558 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.895249 (+ 1691us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.897346 (+ 2097us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.897687 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12777,"thread_start_us":13066,"threads_started":1}
I20260927 13:50:02.895819 27914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260927 13:50:02.899632 27341 generic_service.cc:196] Checking for leaks (request via RPC)
W20260927 13:50:02.902274 27142 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 13:50:02.908250 27142 consensus_peers.cc:487] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 -> Peer cd7597d7a00a4223aabb7168756d8ebd (127.20.133.124:38229): Couldn't send request to peer cd7597d7a00a4223aabb7168756d8ebd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 13:50:02.919101 27773 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cd7597d7a00a4223aabb7168756d8ebd (127.20.133.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:02.923311 27801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: NON_VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: true } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:02.923686 27801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: This master's current role is: LEARNER
I20260927 13:50:02.924970 27913 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2: attempting to promote NON_VOTER 858b03dc507a47ea9997ce955053284a to VOTER
I20260927 13:50:02.925973 27913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } }
W20260927 13:50:02.929105 27142 consensus_peers.cc:487] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 -> Peer cd7597d7a00a4223aabb7168756d8ebd (127.20.133.124:38229): Couldn't send request to peer cd7597d7a00a4223aabb7168756d8ebd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 13:50:02.929132 27773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 LEARNER]: Refusing update from remote peer 25826f702bd249f280d27611d1ce1ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 13:50:02.929829 27913 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 13:50:02.932358 27914 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 858b03dc507a47ea9997ce955053284a (127.20.133.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:02.933113 27773 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 858b03dc507a47ea9997ce955053284a (127.20.133.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:02.934757 27718 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:02.935166 27718 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:02.935359 27718 tablet_replica.cc:330] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: stopping tablet replica
I20260927 13:50:02.937165 27801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:02.937434 27801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:50:02.945274 27718 master.cc:552] Master@127.20.133.124:38229 shutdown complete.
I20260927 13:50:02.945427 27718 master_runner.cc:305] Clearing existing system tablet
I20260927 13:50:02.945865 27718 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:02.946000 27718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:02.948709 27913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:02.949014 27913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:02.949371 27913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:02.949613 27913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:02.949990 27718 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 13:50:02.951665 27919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:02.952157 27718 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:02.952661 27718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 13:50:02.952822 27718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/wal
uuid: "cd7597d7a00a4223aabb7168756d8ebd"
format_stamp: "Formatted at 2026-09-27 13:50:02 on dist-test-slave-q25r"
server_key: "ce65cac7f22ff5575b117efd2213fdea"
server_key_iv: "df6d7ca57b29a87c551e77123f93c966"
server_key_version: "encryptionkey@0"
I20260927 13:50:02.958720 27718 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:02.966701 27718 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 13:50:02.967026 27718 log.cc:1196] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 13:50:02.967629 27718 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: Deleting consensus metadata
I20260927 13:50:02.967945 27718 master_runner.cc:315] Copying system tablet from 127.20.133.126:33783
I20260927 13:50:02.970629 27718 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: tablet copy: Beginning tablet copy session from remote peer at address 127.20.133.126:33783
I20260927 13:50:02.986214 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.971791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.20.133.126:33783)
0927 13:50:02.972016 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:02.972023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:02.972139 (+ 116us) server_negotiation.cc:407] Connection header received
0927 13:50:02.972443 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:02.972449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:02.972525 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:02.972591 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:02.973787 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.974894 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.976770 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.977446 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.977570 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.979879 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:02.979886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:02.979891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:02.979930 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:02.983296 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.984129 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.984135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.984139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.984302 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:02.984728 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:02.984733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:02.984737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:02.985069 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:02.985218 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:02.985635 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 13:50:02.985868 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 13:50:02.986289 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:02.971479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:43958)
0927 13:50:02.972029 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:02.972053 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:02.972264 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:02.972806 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:02.972811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:02.973344 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:02.973608 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.973621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.975038 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.975045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.976612 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:02.976630 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:02.977609 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:02.977616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:02.977799 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:02.978545 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:02.978571 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:02.979663 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:02.983426 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.983432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.983436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.983955 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.984442 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:02.984446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:02.984451 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:02.984625 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:02.985180 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:02.985186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:02.985378 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 13:50:02.985530 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 13:50:02.985852 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":554,"threads_started":1}
I20260927 13:50:02.987147 27177 tablet_copy_service.cc:140] P 25826f702bd249f280d27611d1ce1ad2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cd7597d7a00a4223aabb7168756d8ebd ({username='oryx', principal='oryx/127.20.133.124@KRBTEST.COM'} at 127.0.0.1:43958)
I20260927 13:50:02.987318 27177 tablet_copy_service.cc:161] P 25826f702bd249f280d27611d1ce1ad2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cd7597d7a00a4223aabb7168756d8ebd at {username='oryx', principal='oryx/127.20.133.124@KRBTEST.COM'} at 127.0.0.1:43958: session id = cd7597d7a00a4223aabb7168756d8ebd-00000000000000000000000000000000
I20260927 13:50:02.991001 27177 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 13:50:02.992851 27718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 13:50:03.000066 27718 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: tablet copy: Starting download of 0 data blocks...
I20260927 13:50:03.000434 27718 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: tablet copy: Starting download of 1 WAL segments...
I20260927 13:50:03.003106 27718 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 13:50:03.007392 27177 tablet_copy_service.cc:342] P 25826f702bd249f280d27611d1ce1ad2: Request end of tablet copy session cd7597d7a00a4223aabb7168756d8ebd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.133.124@KRBTEST.COM'} at 127.0.0.1:43958
I20260927 13:50:03.007539 27177 tablet_copy_service.cc:433] P 25826f702bd249f280d27611d1ce1ad2: ending tablet copy session cd7597d7a00a4223aabb7168756d8ebd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cd7597d7a00a4223aabb7168756d8ebd
I20260927 13:50:03.019749 27718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:03.022526 27926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:03.023941 27927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:03.031207 27929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:03.039696 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 27218
I20260927 13:50:03.044582 27718 server_base.cc:1034] running on GCE node
I20260927 13:50:03.045020 27718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:03.045717 27718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:03.046939 27718 hybrid_clock.cc:648] HybridClock initialized: now 1790517003046882 us; error 58 us; skew 500 ppm
I20260927 13:50:03.047288 27718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:03 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517003, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:03.049198 27718 init.cc:377] Logged in from keytab as oryx/127.20.133.124@KRBTEST.COM (short username oryx)
I20260927 13:50:03.057554 27718 webserver.cc:460] Webserver started at http://127.20.133.124:42187/ using document root <none> and password file <none>
I20260927 13:50:03.057946 27718 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:03.058044 27718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:03.060678 27718 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 13:50:03.061460 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:38349
--local_ip_for_outbound_sockets=127.20.133.65
--tserver_master_addrs=127.20.133.126:33783
--webserver_port=35185
--webserver_interface=127.20.133.65
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab
--principal=oryx/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.133.126:33783,127.20.133.125:33637
--tserver_master_addrs=127.20.133.126:33783,127.20.133.125:33637,127.20.133.124:38229 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:50:03.062400 27936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:03.062817 27718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 13:50:03.062937 27718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/wal
uuid: "cd7597d7a00a4223aabb7168756d8ebd"
format_stamp: "Formatted at 2026-09-27 13:50:02 on dist-test-slave-q25r"
server_key: "ce65cac7f22ff5575b117efd2213fdea"
server_key_iv: "df6d7ca57b29a87c551e77123f93c966"
server_key_version: "encryptionkey@0"
I20260927 13:50:03.063119 27718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:03.080828 27718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:03.084242 27718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:03.113338 27718 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.124:38229
I20260927 13:50:03.113488 27988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.124:38229 every 8 connection(s)
I20260927 13:50:03.114234 27718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
I20260927 13:50:03.119596 27989 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:50:03.123914 27989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: Bootstrap starting.
I20260927 13:50:03.141481 27989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:50:03.141947 27989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: Bootstrap complete.
I20260927 13:50:03.142817 27989 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } }
I20260927 13:50:03.143052 27989 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd7597d7a00a4223aabb7168756d8ebd, State: Initialized, Role: LEARNER
I20260927 13:50:03.143311 27989 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } }
I20260927 13:50:03.144841 27992 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:03.145169 27989 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:50:03.145315 27992 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: This master's current role is: LEARNER
I20260927 13:50:03.166675 28003 catalog_manager.cc:1259] Loaded cluster ID: a47906cfaa1d4b80bc3aac7116ef0a20
I20260927 13:50:03.166785 28003 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: loading cluster ID for follower catalog manager: success
I20260927 13:50:03.169329 28003 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: acquiring CA information for follower catalog manager: success
I20260927 13:50:03.171149 28003 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 13:50:03.489004 27941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:03.489328 27941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:03.489462 27941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:03.497190 27941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:03.497313 27941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:03.497426 27941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:50:03.507550 27941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab
--principal=oryx/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:38349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=35185
--tserver_master_addrs=127.20.133.126:33783,127.20.133.125:33637,127.20.133.124:38229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:03.508846 27941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:03.510121 27941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 13:50:03.516451 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:03.510882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:38229 (local address 127.0.0.1:52390)
0927 13:50:03.511309 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:03.511330 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:03.511469 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:03.511947 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:03.511952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:03.511980 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:03.512154 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:03.512163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.513422 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.513428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:03.514537 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:03.514549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.515628 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.515634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:03.515812 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:03.515846 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:50:03.515935 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 13:50:03.516079 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":225,"threads_started":1}
I20260927 13:50:03.516467 28008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:03.511033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.20.133.124:38229)
0927 13:50:03.511647 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:03.511651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:03.511667 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:50:03.511728 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:03.511732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:03.511800 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:03.511901 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:03.512295 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.513283 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.514684 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.515468 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.515624 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:03.515826 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:03.515966 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 13:50:03.516097 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":338,"threads_started":1}
I20260927 13:50:03.517506 27964 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 13:50:03.518278 27992 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25826f702bd249f280d27611d1ce1ad2. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:03.518544 27992 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: This master's current role is: LEARNER
W20260927 13:50:03.523805 28011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:03.523824 28010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:03.524489 28013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:03.524652 27941 server_base.cc:1034] running on GCE node
I20260927 13:50:03.525243 27941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:03.526134 27941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:03.527336 27941 hybrid_clock.cc:648] HybridClock initialized: now 1790517003527278 us; error 51 us; skew 500 ppm
I20260927 13:50:03.527637 27941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:03 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517003, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:03.530728 27941 init.cc:377] Logged in from keytab as oryx/127.20.133.65@KRBTEST.COM (short username oryx)
I20260927 13:50:03.532122 27941 webserver.cc:460] Webserver started at http://127.20.133.65:35185/ using document root <none> and password file <none>
I20260927 13:50:03.532935 27941 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:03.533066 27941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:03.538296 27941 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 13:50:03.541029 28020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:03.542351 27941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:50:03.542501 27941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "26bd1ea161b74c5aab11c5a5d20a2e43"
format_stamp: "Formatted at 2026-09-27 13:50:00 on dist-test-slave-q25r"
server_key: "548a2e926683ff55137189c7e3be1e81"
server_key_iv: "199193dc38caf646e7c97efcbc1504ff"
server_key_version: "encryptionkey@0"
I20260927 13:50:03.543444 27941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:03.556097 27941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:03.560384 27941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:03.560675 27941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:03.561630 27941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:03.563081 27941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:03.563185 27941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:03.563299 27941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:03.563369 27941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:03.593827 27941 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:38349
I20260927 13:50:03.593883 28132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:38349 every 8 connection(s)
I20260927 13:50:03.595289 27941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:50:03.600092 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 27941
I20260927 13:50:03.600488 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 27352
Sep 27 13:50:03 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517003, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
Sep 27 13:50:03 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517003, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.124@KRBTEST.COM
Sep 27 13:50:03 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.133.66@KRBTEST.COM
Sep 27 13:50:03 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517003, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.125@KRBTEST.COM
I20260927 13:50:03.661878 28008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:03.598734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:48735 (local address 127.20.133.124:38229)
0927 13:50:03.598928 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:03.598935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:03.639681 (+ 40746us) server_negotiation.cc:407] Connection header received
0927 13:50:03.639837 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:03.639844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:03.639947 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:03.640070 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:03.643458 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.644560 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.646872 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.647479 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.647603 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:03.654297 (+ 6694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:03.654305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:03.654309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:03.654350 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:03.657731 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:03.658932 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:03.658937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:03.658941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:03.659122 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:03.659630 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:03.659635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:03.659639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:03.660015 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:03.660162 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:03.661283 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0927 13:50:03.661533 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:50:03.663393 28140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:03.598873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:38229 (local address 127.20.133.65:48735)
0927 13:50:03.639526 (+ 40653us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:03.639551 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:03.639716 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:03.640704 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:03.640715 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:03.641894 (+ 1179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:03.643171 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:03.643211 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.644792 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.644796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:03.646709 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:03.646724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.647918 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.647923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:03.648092 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:03.650931 (+ 2839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:03.650963 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:03.654050 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:03.657901 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:03.657912 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:03.657933 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:03.658787 (+ 854us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:03.659229 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:03.659233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:03.659236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:03.659489 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:03.660257 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:03.660266 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:03.660831 (+ 565us) client_negotiation.cc:769] Sending connection context
0927 13:50:03.662154 (+ 1323us) client_negotiation.cc:240] Negotiation successful
0927 13:50:03.662456 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40468,"mutex_wait_us":540,"spinlock_wait_cycles":25088,"thread_start_us":28345,"threads_started":1}
I20260927 13:50:03.663673 28138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:03.598298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:59151 (local address 127.20.133.126:33783)
0927 13:50:03.640183 (+ 41885us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:03.640189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:03.640209 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:50:03.640257 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:03.640260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:03.640316 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:03.640442 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:03.643459 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.644551 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.646896 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.647410 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.647519 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:03.654083 (+ 6564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:03.654091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:03.654096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:03.654140 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:03.657718 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:03.658932 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:03.658937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:03.658941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:03.659321 (+ 380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:03.659736 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:03.659741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:03.659744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:03.660067 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:03.660195 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:03.663055 (+ 2860us) server_negotiation.cc:299] Negotiation successful
0927 13:50:03.663329 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41750,"thread_start_us":41641,"threads_started":1}
I20260927 13:50:03.664024 28141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:03.598871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.65:59151)
0927 13:50:03.624269 (+ 25398us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:03.624359 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:03.627033 (+ 2674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:03.640699 (+ 13666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:03.640714 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:03.641895 (+ 1181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:03.643171 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:03.643211 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.644767 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.644772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:03.646763 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:03.646774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.647639 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.647647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:03.647944 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:03.650931 (+ 2987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:03.650964 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:03.653733 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:03.657901 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:03.657912 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:03.657933 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:03.658787 (+ 854us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:03.659452 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:03.659457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:03.659461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:03.659618 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:03.660257 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:03.660266 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:03.661213 (+ 947us) client_negotiation.cc:769] Sending connection context
0927 13:50:03.661356 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 13:50:03.661903 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24964,"mutex_wait_us":943,"spinlock_wait_cycles":41472,"thread_start_us":30227,"threads_started":1}
I20260927 13:50:03.664160 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:03.598277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:60421 (local address 127.20.133.125:33637)
0927 13:50:03.643978 (+ 45701us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:03.643984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:03.644010 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:50:03.644069 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:03.644452 (+ 383us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:03.644550 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:03.644666 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:03.646440 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.647562 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.649200 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.649636 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.649736 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:03.656642 (+ 6906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:03.656654 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:03.656659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:03.656706 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:03.660378 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:03.661102 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:03.661107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:03.661110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:03.661244 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:03.661680 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:03.661684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:03.661687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:03.662018 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:03.662172 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:03.663433 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0927 13:50:03.663650 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45568,"thread_start_us":45429,"threads_started":1}
I20260927 13:50:03.664533 28139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:03.599019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:33637 (local address 127.20.133.65:60421)
0927 13:50:03.628988 (+ 29969us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:03.629012 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:03.629157 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:03.644912 (+ 15755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:03.644918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:03.645821 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:03.646245 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:03.646258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.647769 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.647773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:03.649043 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:03.649055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.649803 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.649808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:03.649923 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:03.653737 (+ 3814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:03.653761 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:03.656301 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:03.660476 (+ 4175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:03.660480 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:03.660483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:03.660965 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:03.661381 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:03.661386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:03.661389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:03.661554 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:03.662234 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:03.662239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:03.662413 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 13:50:03.662539 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 13:50:03.662879 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29813,"mutex_wait_us":1,"spinlock_wait_cycles":60800,"thread_start_us":40088,"threads_started":1}
I20260927 13:50:03.665179 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:03.601887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.20.133.66:46135)
0927 13:50:03.632915 (+ 31028us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:03.632925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:03.633747 (+ 822us) server_negotiation.cc:407] Connection header received
0927 13:50:03.633853 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:03.633865 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:03.634182 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:03.634372 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:03.636964 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.638774 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.642359 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:03.643249 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:03.643387 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:03.655490 (+ 12103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:03.655511 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:03.655530 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:03.655582 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:03.659523 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:03.660531 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:03.660540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:03.660544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:03.660697 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:03.661079 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:03.661084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:03.661087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:03.661524 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:03.661836 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:03.663376 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0927 13:50:03.664072 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30847,"thread_start_us":30663,"threads_started":1}
I20260927 13:50:03.665726 28133 heartbeater.cc:347] Connected to a master server at 127.20.133.124:38229
I20260927 13:50:03.666006 28133 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:03.666682 28133 heartbeater.cc:511] Master 127.20.133.124:38229 requested a full tablet report, sending...
I20260927 13:50:03.667348 28135 heartbeater.cc:347] Connected to a master server at 127.20.133.125:33637
I20260927 13:50:03.667469 28135 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:03.667917 28135 heartbeater.cc:511] Master 127.20.133.125:33637 requested a full tablet report, sending...
I20260927 13:50:03.668571 27475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:03.668808 28134 heartbeater.cc:347] Connected to a master server at 127.20.133.126:33783
I20260927 13:50:03.668918 28134 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:03.668936 27954 ts_manager.cc:194] Registered new tserver with Master: 26bd1ea161b74c5aab11c5a5d20a2e43 (127.20.133.65:38349)
I20260927 13:50:03.669303 28134 heartbeater.cc:511] Master 127.20.133.126:33783 requested a full tablet report, sending...
I20260927 13:50:03.669618 27763 ts_manager.cc:194] Registered new tserver with Master: 26bd1ea161b74c5aab11c5a5d20a2e43 (127.20.133.65:38349)
I20260927 13:50:03.670471 27156 ts_manager.cc:194] Re-registered known tserver with Master: 26bd1ea161b74c5aab11c5a5d20a2e43 (127.20.133.65:38349)
I20260927 13:50:03.671607 27156 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:59151
I20260927 13:50:03.764199 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 27352
I20260927 13:50:03.781914 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:46135
--local_ip_for_outbound_sockets=127.20.133.66
--tserver_master_addrs=127.20.133.126:33783
--webserver_port=43741
--webserver_interface=127.20.133.66
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab
--principal=oryx/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.133.126:33783,127.20.133.125:33637
--tserver_master_addrs=127.20.133.126:33783,127.20.133.125:33637,127.20.133.124:38229 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:04.109176 28145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:04.109499 28145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:04.109642 28145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:04.119650 28145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:04.119801 28145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:04.119992 28145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:50:04.127107 27914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: NON_VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: true } } }
I20260927 13:50:04.127482 27914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:04.129254 28150 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2: attempting to promote NON_VOTER cd7597d7a00a4223aabb7168756d8ebd to VOTER
I20260927 13:50:04.129951 28150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } }
I20260927 13:50:04.132184 27773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 FOLLOWER]: Refusing update from remote peer 25826f702bd249f280d27611d1ce1ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 13:50:04.132279 27964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 LEARNER]: Refusing update from remote peer 25826f702bd249f280d27611d1ce1ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 13:50:04.132898 27914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 13:50:04.133251 28149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 13:50:04.134887 28145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab
--principal=oryx/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:46135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=43741
--tserver_master_addrs=127.20.133.126:33783,127.20.133.125:33637,127.20.133.124:38229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:04.136224 27914 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cd7597d7a00a4223aabb7168756d8ebd (127.20.133.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:04.136636 28145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:04.137356 27773 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cd7597d7a00a4223aabb7168756d8ebd (127.20.133.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:04.138375 28145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 13:50:04.139398 28149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:04.139688 28149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:04.140028 28149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:04.139994 27914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:04.140261 28149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:04.140254 27914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:04.141470 27964 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cd7597d7a00a4223aabb7168756d8ebd (127.20.133.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:04.148862 28155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:04.149821 28156 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25826f702bd249f280d27611d1ce1ad2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:04.150269 28156 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:50:04.150506 28155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: This master's current role is: FOLLOWER
W20260927 13:50:04.157188 28159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:04.157132 28161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:04.157197 28158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:04.158344 28145 server_base.cc:1034] running on GCE node
I20260927 13:50:04.158957 28145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:04.159904 28145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:04.161091 28145 hybrid_clock.cc:648] HybridClock initialized: now 1790517004161034 us; error 49 us; skew 500 ppm
I20260927 13:50:04.161376 28145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:04 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517004, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:04.164705 28145 init.cc:377] Logged in from keytab as oryx/127.20.133.66@KRBTEST.COM (short username oryx)
I20260927 13:50:04.166147 28145 webserver.cc:460] Webserver started at http://127.20.133.66:43741/ using document root <none> and password file <none>
I20260927 13:50:04.166810 28145 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:04.166936 28145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:04.172757 28145 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:50:04.175472 28168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:04.176873 28145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:50:04.177034 28145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "be53d36199114b9fbb19b95c28e8ee48"
format_stamp: "Formatted at 2026-09-27 13:50:00 on dist-test-slave-q25r"
server_key: "efda4566e804acdf4d6e687ebbe89394"
server_key_iv: "8f3d63a38b232126cf401c4110a8f60f"
server_key_version: "encryptionkey@0"
I20260927 13:50:04.178033 28145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:04.195307 28145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:04.199520 28145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:04.199797 28145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:04.200816 28145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:04.202196 28145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:04.202292 28145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:04.202400 28145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:04.202476 28145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:04.234701 28145 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:46135
I20260927 13:50:04.234750 28280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:46135 every 8 connection(s)
I20260927 13:50:04.236186 28145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:50:04.243242 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 28145
I20260927 13:50:04.243563 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 27485
Sep 27 13:50:04 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517004, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
Sep 27 13:50:04 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517004, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.125@KRBTEST.COM
Sep 27 13:50:04 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517004, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.124@KRBTEST.COM
Sep 27 13:50:04 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.133.67@KRBTEST.COM
I20260927 13:50:04.292927 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.239640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:43421 (local address 127.20.133.125:33637)
0927 13:50:04.252044 (+ 12404us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:04.252053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:04.272917 (+ 20864us) server_negotiation.cc:407] Connection header received
0927 13:50:04.273178 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:04.273183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:04.273284 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:04.273406 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:04.275731 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.276627 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.279077 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.279660 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.279757 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.285042 (+ 5285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:04.285049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:04.285054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:04.285099 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:04.288717 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.289950 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.289955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.289959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.290104 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.290683 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.290688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.290692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.291055 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:04.291186 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:04.292322 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0927 13:50:04.292580 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12200,"thread_start_us":12432,"threads_started":1}
I20260927 13:50:04.293040 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.239383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:33855 (local address 127.20.133.126:33783)
0927 13:50:04.252691 (+ 13308us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:04.252698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:04.268753 (+ 16055us) server_negotiation.cc:407] Connection header received
0927 13:50:04.271044 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:04.271050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:04.271162 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:04.271283 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:04.276006 (+ 4723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.277050 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.279041 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.279533 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.279641 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.285569 (+ 5928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:04.285574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:04.285577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:04.285621 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:04.289140 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.290206 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.290213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.290216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.290384 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.291083 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.291090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.291093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.291596 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:04.291809 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:04.292441 (+ 632us) server_negotiation.cc:299] Negotiation successful
0927 13:50:04.292688 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13038,"thread_start_us":12944,"threads_started":1}
I20260927 13:50:04.293826 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.255970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:57153 (local address 127.20.133.124:38229)
0927 13:50:04.269507 (+ 13537us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:04.269513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:04.274764 (+ 5251us) server_negotiation.cc:407] Connection header received
0927 13:50:04.274931 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:04.274936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:04.275020 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:04.275138 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:04.276230 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.277351 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.279251 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.279849 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.279964 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.285454 (+ 5490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:04.285461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:04.285465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:04.285514 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:04.288790 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.290066 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.290071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.290074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.290216 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.290683 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.290688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.290692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.292225 (+ 1533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:04.292391 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:04.293226 (+ 835us) server_negotiation.cc:299] Negotiation successful
0927 13:50:04.293455 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13391,"thread_start_us":15046,"threads_started":1}
I20260927 13:50:04.294999 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.244603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:33637 (local address 127.20.133.66:43421)
0927 13:50:04.272809 (+ 28206us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:04.272833 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:04.273033 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:04.273704 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:04.273707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:04.274433 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:04.275509 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.275539 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.276778 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.276781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.278918 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.278927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.280046 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.280053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.280234 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.281254 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:04.281273 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:04.284762 (+ 3489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:04.288939 (+ 4177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.288949 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.288976 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.289801 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.290258 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.290262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.290266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.290558 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.291352 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:04.291361 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:04.291833 (+ 472us) client_negotiation.cc:769] Sending connection context
0927 13:50:04.293166 (+ 1333us) client_negotiation.cc:240] Negotiation successful
0927 13:50:04.293529 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28046,"mutex_wait_us":142,"spinlock_wait_cycles":45696,"thread_start_us":27838,"threads_started":1}
I20260927 13:50:04.295038 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.256085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:38229 (local address 127.20.133.66:57153)
0927 13:50:04.274651 (+ 18566us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:04.274667 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:04.274810 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:04.275254 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:04.275259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:04.275765 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:04.276069 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.276081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.277511 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.277517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.279112 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.279127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.280234 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.280242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.280476 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.280980 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:04.281007 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:04.285217 (+ 4210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:04.288939 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.288949 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.288976 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.289934 (+ 958us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.290329 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.290334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.290337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.290558 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.292449 (+ 1891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:04.292453 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:04.292589 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 13:50:04.292758 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 13:50:04.294172 (+ 1414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18423,"mutex_wait_us":1,"spinlock_wait_cycles":4864,"thread_start_us":11888,"threads_started":1}
I20260927 13:50:04.295094 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.240161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.66:33855)
0927 13:50:04.268523 (+ 28362us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:04.268589 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:04.270707 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:04.273207 (+ 2500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:04.273220 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:04.275024 (+ 1804us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:04.275849 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.275865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.277192 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.277197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.278911 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.278925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.279774 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.279782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.280060 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.280899 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:04.280941 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:04.285378 (+ 4437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:04.289446 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.289452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.289456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.290058 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.290596 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.290600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.290604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.290790 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.291942 (+ 1152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:04.291946 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:04.292143 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 13:50:04.293296 (+ 1153us) client_negotiation.cc:240] Negotiation successful
0927 13:50:04.293692 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27964,"mutex_wait_us":117,"thread_start_us":34080,"threads_started":1}
I20260927 13:50:04.296646 28281 heartbeater.cc:347] Connected to a master server at 127.20.133.124:38229
I20260927 13:50:04.296675 28282 heartbeater.cc:347] Connected to a master server at 127.20.133.126:33783
I20260927 13:50:04.296977 28281 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:04.296974 28282 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:04.297827 28283 heartbeater.cc:347] Connected to a master server at 127.20.133.125:33637
I20260927 13:50:04.297859 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.245664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.20.133.67:44205)
0927 13:50:04.271618 (+ 25954us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:04.271624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:04.272131 (+ 507us) server_negotiation.cc:407] Connection header received
0927 13:50:04.272299 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:04.272308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:04.272631 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:04.274845 (+ 2214us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:04.275111 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.277901 (+ 2790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.280551 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.281375 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.281504 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.288451 (+ 6947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:04.288469 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:04.288490 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:04.288538 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:04.292422 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.293825 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.293834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.293837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.294306 (+ 469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.294753 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.294758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.294761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.295893 (+ 1132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:04.296167 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:04.297268 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0927 13:50:04.297523 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25808,"thread_start_us":25665,"threads_started":1}
I20260927 13:50:04.297907 28282 heartbeater.cc:511] Master 127.20.133.126:33783 requested a full tablet report, sending...
I20260927 13:50:04.297957 28283 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:04.298220 28281 heartbeater.cc:511] Master 127.20.133.124:38229 requested a full tablet report, sending...
I20260927 13:50:04.298389 28283 heartbeater.cc:511] Master 127.20.133.125:33637 requested a full tablet report, sending...
I20260927 13:50:04.299124 27608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:04.299539 27156 ts_manager.cc:194] Re-registered known tserver with Master: be53d36199114b9fbb19b95c28e8ee48 (127.20.133.66:46135)
I20260927 13:50:04.299721 27763 ts_manager.cc:194] Registered new tserver with Master: be53d36199114b9fbb19b95c28e8ee48 (127.20.133.66:46135)
I20260927 13:50:04.300549 27954 ts_manager.cc:194] Registered new tserver with Master: be53d36199114b9fbb19b95c28e8ee48 (127.20.133.66:46135)
I20260927 13:50:04.300566 27156 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:33855
I20260927 13:50:04.384459 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 27485
I20260927 13:50:04.397708 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:44205
--local_ip_for_outbound_sockets=127.20.133.67
--tserver_master_addrs=127.20.133.126:33783
--webserver_port=36235
--webserver_interface=127.20.133.67
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab
--principal=oryx/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.133.126:33783,127.20.133.125:33637
--tserver_master_addrs=127.20.133.126:33783,127.20.133.125:33637,127.20.133.124:38229 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:50:04.674049 28134 heartbeater.cc:503] Master 127.20.133.126:33783 was elected leader, sending a full tablet report...
W20260927 13:50:04.733121 28294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:04.733438 28294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:04.733580 28294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:04.741173 28294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:04.741303 28294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:04.741448 28294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:50:04.751981 28294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab
--principal=oryx/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:44205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=36235
--tserver_master_addrs=127.20.133.126:33783,127.20.133.125:33637,127.20.133.124:38229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:04.753415 28294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:04.754769 28294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:04.768589 28299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:04.768532 28302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:04.768532 28300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:04.769573 28294 server_base.cc:1034] running on GCE node
I20260927 13:50:04.770262 28294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:04.771311 28294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:04.772538 28294 hybrid_clock.cc:648] HybridClock initialized: now 1790517004772471 us; error 67 us; skew 500 ppm
I20260927 13:50:04.772888 28294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:04 dist-test-slave-q25r krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517004, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:04.781473 28294 init.cc:377] Logged in from keytab as oryx/127.20.133.67@KRBTEST.COM (short username oryx)
I20260927 13:50:04.783193 28294 webserver.cc:460] Webserver started at http://127.20.133.67:36235/ using document root <none> and password file <none>
I20260927 13:50:04.783957 28294 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:04.784106 28294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:04.789930 28294 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:50:04.792784 28309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:04.794240 28294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:50:04.794394 28294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "9e26d452393e4256b18196c585186f24"
format_stamp: "Formatted at 2026-09-27 13:50:01 on dist-test-slave-q25r"
server_key: "d8772a152c76d941820dee5614f220ab"
server_key_iv: "7d77fc6f1fdc212ca9008813ad73f030"
server_key_version: "encryptionkey@0"
I20260927 13:50:04.795529 28294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:04.816282 28294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:04.821106 28294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:04.821394 28294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:04.822435 28294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:04.823912 28294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:04.824002 28294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:04.824103 28294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:04.824177 28294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:04.855839 28294 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:44205
I20260927 13:50:04.855891 28421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:44205 every 8 connection(s)
I20260927 13:50:04.857384 28294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:50:04.858656 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 28294
Sep 27 13:50:04 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517004, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.124@KRBTEST.COM
I20260927 13:50:04.891011 28431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.862132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:42549 (local address 127.20.133.124:38229)
0927 13:50:04.867895 (+ 5763us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:04.867900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:04.867917 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:50:04.867970 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:04.867974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:04.868047 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:04.868150 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:04.870743 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.872011 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.874778 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.875503 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.875656 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.881434 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:04.881443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:04.881448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:04.881499 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:04.885712 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.887006 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.887014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.887019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.887214 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.887988 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.887993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.888068 (+ 75us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.888704 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:04.890130 (+ 1426us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:04.890294 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 13:50:04.890582 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5655,"thread_start_us":5546,"threads_started":1}
I20260927 13:50:04.893929 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.861439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:38229 (local address 127.20.133.67:42549)
0927 13:50:04.863907 (+ 2468us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:04.863967 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:04.866661 (+ 2694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:04.868527 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:04.868542 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:04.869540 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:04.870489 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.870515 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.872246 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.872253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.874584 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.874601 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.876015 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.876023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.876415 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.877351 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:04.877387 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:04.881034 (+ 3647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:04.886008 (+ 4974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.886021 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.886045 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.886820 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.887357 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.887363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.887367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.887833 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.888990 (+ 1157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:04.889000 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:04.889586 (+ 586us) client_negotiation.cc:769] Sending connection context
0927 13:50:04.889919 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 13:50:04.892159 (+ 2240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2150,"thread_start_us":1887,"threads_started":1}
I20260927 13:50:04.896842 28422 heartbeater.cc:347] Connected to a master server at 127.20.133.124:38229
I20260927 13:50:04.897249 28422 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:04.898360 28422 heartbeater.cc:511] Master 127.20.133.124:38229 requested a full tablet report, sending...
I20260927 13:50:04.900758 27954 ts_manager.cc:194] Registered new tserver with Master: 9e26d452393e4256b18196c585186f24 (127.20.133.67:44205)
Sep 27 13:50:04 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517004, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
Sep 27 13:50:04 dist-test-slave-q25r krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517004, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.125@KRBTEST.COM
I20260927 13:50:04.926060 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.862267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.67:45863)
0927 13:50:04.905491 (+ 43224us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:04.905514 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:04.905663 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:04.907285 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:04.907289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:04.909143 (+ 1854us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:04.909516 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.909535 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.910991 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.910996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.912460 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.912475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.913776 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.913783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.913968 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.914559 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:04.914581 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:04.918003 (+ 3422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:04.922880 (+ 4877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.922886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.922890 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.923492 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.923931 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.923936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.923940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.924109 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.924796 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:04.924801 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:04.924977 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 13:50:04.925104 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 13:50:04.925431 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43077,"thread_start_us":43646,"threads_started":1}
I20260927 13:50:04.926774 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.861111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:45863 (local address 127.20.133.126:33783)
0927 13:50:04.904905 (+ 43794us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:04.904913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:04.906358 (+ 1445us) server_negotiation.cc:407] Connection header received
0927 13:50:04.906449 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:04.906455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:04.906652 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:04.906788 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:04.909699 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.910824 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.912842 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.913568 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.913694 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.918352 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:04.918361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:04.918367 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:04.918422 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:04.922703 (+ 4281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.923642 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.923650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.923654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.923818 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.924221 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.924226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.924230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.924684 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:04.924849 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:04.925365 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 13:50:04.925643 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43624,"thread_start_us":43497,"threads_started":1}
I20260927 13:50:04.933995 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.861620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:33637 (local address 127.20.133.67:43667)
0927 13:50:04.902404 (+ 40784us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:04.902430 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:04.902583 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:04.907024 (+ 4441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:04.907030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:04.907926 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:04.908142 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.908156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.909522 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.909526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.910939 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.910953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.911970 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.911976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:04.912223 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.914202 (+ 1979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:04.914232 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:04.919082 (+ 4850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:04.930698 (+ 11616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.930706 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.930711 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.931328 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.931870 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:04.931877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:04.931881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:04.932052 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:04.932797 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:04.932803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:04.933001 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 13:50:04.933169 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 13:50:04.933537 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40637,"spinlock_wait_cycles":68864,"thread_start_us":43465,"threads_started":1}
I20260927 13:50:04.935688 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.860789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:43667 (local address 127.20.133.125:33637)
0927 13:50:04.906490 (+ 45701us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:04.906497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:04.906519 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:50:04.906588 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:04.906593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:04.906676 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:04.906789 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:04.908314 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.909363 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.911071 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.911783 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.914993 (+ 3210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.923040 (+ 8047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:04.923048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:04.923054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:04.923108 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:04.930448 (+ 7340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.931532 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.931539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.931543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.931727 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.932209 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.932214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.932218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.932625 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:04.934779 (+ 2154us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:04.934943 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 13:50:04.935229 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45562,"thread_start_us":45396,"threads_started":1}
I20260927 13:50:04.936908 28424 heartbeater.cc:347] Connected to a master server at 127.20.133.125:33637
I20260927 13:50:04.937081 28424 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:04.937582 28424 heartbeater.cc:511] Master 127.20.133.125:33637 requested a full tablet report, sending...
I20260927 13:50:04.939128 27763 ts_manager.cc:194] Registered new tserver with Master: 9e26d452393e4256b18196c585186f24 (127.20.133.67:44205)
I20260927 13:50:04.945536 28423 heartbeater.cc:347] Connected to a master server at 127.20.133.126:33783
I20260927 13:50:04.945737 28423 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:04.946323 28423 heartbeater.cc:511] Master 127.20.133.126:33783 requested a full tablet report, sending...
I20260927 13:50:04.947556 27156 ts_manager.cc:194] Re-registered known tserver with Master: 9e26d452393e4256b18196c585186f24 (127.20.133.67:44205)
I20260927 13:50:04.948755 27156 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:45863
I20260927 13:50:04.965083 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.913084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.20.133.125:33637)
0927 13:50:04.940014 (+ 26930us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:04.940021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:04.948206 (+ 8185us) server_negotiation.cc:407] Connection header received
0927 13:50:04.948376 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:04.948382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:04.948483 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:04.948591 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:04.951267 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.952282 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.954030 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.954629 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.954755 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.957722 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:04.957727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:04.957731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:04.957768 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:04.961239 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.962439 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.962444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.962447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.962597 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.963033 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.963038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.963041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.963473 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:04.963794 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:04.964314 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 13:50:04.964707 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26791,"thread_start_us":31814,"threads_started":1}
I20260927 13:50:04.965458 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.913105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.20.133.126:33783)
0927 13:50:04.927266 (+ 14161us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:04.927273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:04.947384 (+ 20111us) server_negotiation.cc:407] Connection header received
0927 13:50:04.947674 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:04.947680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:04.947788 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:04.947879 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:04.950707 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.951756 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.953725 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.954364 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.954471 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.958423 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:04.958430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:04.958436 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:04.958480 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:04.961358 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.963220 (+ 1862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.963226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.963228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.963441 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.963844 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.963851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.963854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.964279 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:04.964595 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:04.964880 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 13:50:04.965137 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":14021,"thread_start_us":31483,"threads_started":1}
I20260927 13:50:04.966463 28431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.912844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.20.133.124:38229)
0927 13:50:04.913389 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:04.913395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:04.948443 (+ 35048us) server_negotiation.cc:407] Connection header received
0927 13:50:04.948516 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:04.948521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:04.948639 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:04.948733 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:04.951241 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.952095 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.953573 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:04.954190 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:04.954331 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:04.957997 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:04.958004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:04.958008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:04.958050 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:04.961257 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.962344 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.962350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.962354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.962549 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:04.963140 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:04.963146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:04.963149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:04.963513 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:04.963779 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:04.965980 (+ 2201us) server_negotiation.cc:299] Negotiation successful
0927 13:50:04.966176 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397}
I20260927 13:50:04.970304 27156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:50:04.977337 27156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:50:05.007335 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.995984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:46135 (local address 127.0.0.1:58774)
0927 13:50:04.997191 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:04.997211 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:04.997341 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:04.998382 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:04.998389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:04.998417 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:04.999225 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:04.999239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.001406 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.001413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:05.003815 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:05.003834 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.006372 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.006380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:05.006587 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.006627 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 13:50:05.006724 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 13:50:05.006908 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"thread_start_us":1393,"threads_started":1}
I20260927 13:50:05.008762 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.996212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.20.133.66:46135)
0927 13:50:04.996698 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:04.996707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:04.997470 (+ 763us) server_negotiation.cc:407] Connection header received
0927 13:50:04.997600 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:04.997610 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:04.997944 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:04.998151 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:04.999431 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.001219 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.004044 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.005802 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.005966 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.008066 (+ 2100us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:05.008169 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 13:50:05.008423 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":262,"threads_started":1}
I20260927 13:50:05.013541 28216 tablet_service.cc:1467] Processing CreateTablet for tablet bb5c9d5cc66749adb47485bb2be0e90e (DEFAULT_TABLE table=test-table [id=7be8418332024e04abe0901f1fbfe0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:05.015435 28216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5c9d5cc66749adb47485bb2be0e90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:05.019199 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.997008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38349 (local address 127.0.0.1:45110)
0927 13:50:05.007796 (+ 10788us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:05.007820 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:05.007945 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:05.010244 (+ 2299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:05.010251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:05.010287 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:05.010491 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:05.010502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.012569 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.012575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:05.014715 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:05.014733 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.018429 (+ 3696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.018436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:05.018621 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.018655 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:50:05.018750 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 13:50:05.018910 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10657,"threads_started":1}
I20260927 13:50:05.019915 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.997245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45110 (local address 127.20.133.65:38349)
0927 13:50:05.009271 (+ 12026us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:05.009282 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:05.009316 (+ 34us) server_negotiation.cc:407] Connection header received
0927 13:50:05.009432 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:05.009442 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:05.009753 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:05.009985 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:05.010686 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.012376 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.014950 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.017964 (+ 3014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.018089 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.018334 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:05.019444 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0927 13:50:05.019623 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11460,"thread_start_us":11717,"threads_started":1}
I20260927 13:50:05.023939 28448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.995321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:44205 (local address 127.0.0.1:35598)
0927 13:50:04.995937 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:04.995957 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:04.996089 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:05.017426 (+ 21337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:05.017431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:05.017463 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:05.017780 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:05.017792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.020014 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.020019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:05.021659 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:05.021672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.023123 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.023130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:05.023332 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.023368 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 13:50:05.023444 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 13:50:05.023581 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":305,"threads_started":1}
I20260927 13:50:05.025007 28449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:04.995607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.20.133.67:44205)
0927 13:50:05.016886 (+ 21279us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:05.016892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:05.016922 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:50:05.017009 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:05.017014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:05.017211 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:05.017402 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:05.018106 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.019840 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.021891 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.022891 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.024185 (+ 1294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.024441 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:05.024520 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 13:50:05.024683 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21104,"thread_start_us":20934,"threads_started":1}
I20260927 13:50:05.026011 28068 tablet_service.cc:1467] Processing CreateTablet for tablet bb5c9d5cc66749adb47485bb2be0e90e (DEFAULT_TABLE table=test-table [id=7be8418332024e04abe0901f1fbfe0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:05.027760 28068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5c9d5cc66749adb47485bb2be0e90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:05.028956 28357 tablet_service.cc:1467] Processing CreateTablet for tablet bb5c9d5cc66749adb47485bb2be0e90e (DEFAULT_TABLE table=test-table [id=7be8418332024e04abe0901f1fbfe0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:05.030460 28357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5c9d5cc66749adb47485bb2be0e90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:05.031086 28454 tablet_bootstrap.cc:492] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: Bootstrap starting.
I20260927 13:50:05.034241 28454 tablet_bootstrap.cc:654] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:05.035614 28454 log.cc:824] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:05.038475 28454 tablet_bootstrap.cc:492] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: No bootstrap required, opened a new log
I20260927 13:50:05.038738 28454 ts_tablet_manager.cc:1397] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260927 13:50:05.045106 28456 tablet_bootstrap.cc:492] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: Bootstrap starting.
I20260927 13:50:05.047354 28454 raft_consensus.cc:348] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } }
I20260927 13:50:05.047849 28454 raft_consensus.cc:374] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:05.047955 28454 raft_consensus.cc:729] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be53d36199114b9fbb19b95c28e8ee48, State: Initialized, Role: FOLLOWER
I20260927 13:50:05.048638 28454 consensus_queue.cc:260] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } }
I20260927 13:50:05.049904 28456 tablet_bootstrap.cc:654] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:05.051491 28456 log.cc:824] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:05.052424 28457 tablet_bootstrap.cc:492] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: Bootstrap starting.
I20260927 13:50:05.055657 28456 tablet_bootstrap.cc:492] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: No bootstrap required, opened a new log
I20260927 13:50:05.055696 28457 tablet_bootstrap.cc:654] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:05.056007 28456 ts_tablet_manager.cc:1397] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 13:50:05.056732 28282 heartbeater.cc:503] Master 127.20.133.126:33783 was elected leader, sending a full tablet report...
I20260927 13:50:05.057026 28457 log.cc:824] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:05.057401 28454 ts_tablet_manager.cc:1428] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
I20260927 13:50:05.060807 28457 tablet_bootstrap.cc:492] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: No bootstrap required, opened a new log
I20260927 13:50:05.061141 28457 ts_tablet_manager.cc:1397] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 13:50:05.070248 28456 raft_consensus.cc:348] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } }
I20260927 13:50:05.070804 28456 raft_consensus.cc:374] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:05.070912 28456 raft_consensus.cc:729] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26bd1ea161b74c5aab11c5a5d20a2e43, State: Initialized, Role: FOLLOWER
I20260927 13:50:05.071621 28456 consensus_queue.cc:260] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } }
I20260927 13:50:05.071574 28457 raft_consensus.cc:348] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } }
I20260927 13:50:05.072027 28457 raft_consensus.cc:374] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:05.072129 28457 raft_consensus.cc:729] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e26d452393e4256b18196c585186f24, State: Initialized, Role: FOLLOWER
I20260927 13:50:05.072840 28457 consensus_queue.cc:260] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } }
I20260927 13:50:05.073768 28456 ts_tablet_manager.cc:1428] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: Time spent starting tablet: real 0.018s user 0.006s sys 0.008s
I20260927 13:50:05.075886 28423 heartbeater.cc:503] Master 127.20.133.126:33783 was elected leader, sending a full tablet report...
I20260927 13:50:05.078995 28457 ts_tablet_manager.cc:1428] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
W20260927 13:50:05.099231 28136 tablet.cc:2369] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:50:05.159494 28425 tablet.cc:2369] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:50:05.176185 28458 raft_consensus.cc:482] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:50:05.176507 28458 raft_consensus.cc:504] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } }
I20260927 13:50:05.178277 28458 leader_election.cc:288] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e26d452393e4256b18196c585186f24 (127.20.133.67:44205), 26bd1ea161b74c5aab11c5a5d20a2e43 (127.20.133.65:38349)
I20260927 13:50:05.186074 28449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:05.178667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:48511 (local address 127.20.133.67:44205)
0927 13:50:05.178986 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:05.178993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:05.179443 (+ 450us) server_negotiation.cc:407] Connection header received
0927 13:50:05.179682 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:05.179687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:05.179754 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:05.179853 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:05.180643 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.181483 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.184097 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.185048 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.185206 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.185274 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:05.185693 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 13:50:05.185833 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260927 13:50:05.186385 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:05.178455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:44205 (local address 127.20.133.66:48511)
0927 13:50:05.179307 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:05.179327 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:05.179499 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:05.179923 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:05.179927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:05.179953 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:05.180457 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:05.180473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.181671 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.181676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:05.183866 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:05.183885 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.185397 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.185402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:05.185530 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.185555 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 13:50:05.185638 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 13:50:05.185783 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":428,"threads_started":1}
I20260927 13:50:05.187119 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:05.178751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:45381 (local address 127.20.133.65:38349)
0927 13:50:05.179153 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:05.179157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:05.180104 (+ 947us) server_negotiation.cc:407] Connection header received
0927 13:50:05.180262 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:05.180267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:05.180499 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:05.180586 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:05.181996 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.183084 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.185215 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.185994 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.186149 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.186221 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:05.186773 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 13:50:05.186890 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260927 13:50:05.187148 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:05.178779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38349 (local address 127.20.133.66:45381)
0927 13:50:05.179989 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:05.180002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:05.180139 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:05.181326 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:05.181331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:05.181359 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:05.181673 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:05.181683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.183303 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.183309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:05.185044 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:05.185058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.186373 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.186382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:05.186560 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.186595 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:50:05.186682 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 13:50:05.186814 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"spinlock_wait_cycles":2304,"thread_start_us":972,"threads_started":1}
I20260927 13:50:05.187582 28377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5c9d5cc66749adb47485bb2be0e90e" candidate_uuid: "be53d36199114b9fbb19b95c28e8ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e26d452393e4256b18196c585186f24" is_pre_election: true
I20260927 13:50:05.188156 28377 raft_consensus.cc:2454] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be53d36199114b9fbb19b95c28e8ee48 in term 0.
I20260927 13:50:05.188258 28088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5c9d5cc66749adb47485bb2be0e90e" candidate_uuid: "be53d36199114b9fbb19b95c28e8ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" is_pre_election: true
I20260927 13:50:05.188838 28088 raft_consensus.cc:2454] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be53d36199114b9fbb19b95c28e8ee48 in term 0.
I20260927 13:50:05.188972 28170 leader_election.cc:302] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e26d452393e4256b18196c585186f24, be53d36199114b9fbb19b95c28e8ee48; no voters:
I20260927 13:50:05.189469 28458 raft_consensus.cc:2781] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:50:05.189630 28458 raft_consensus.cc:482] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:50:05.189729 28458 raft_consensus.cc:3037] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:05.191943 28458 raft_consensus.cc:504] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } }
I20260927 13:50:05.192569 28458 leader_election.cc:288] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9e26d452393e4256b18196c585186f24 (127.20.133.67:44205), 26bd1ea161b74c5aab11c5a5d20a2e43 (127.20.133.65:38349)
I20260927 13:50:05.193079 28377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5c9d5cc66749adb47485bb2be0e90e" candidate_uuid: "be53d36199114b9fbb19b95c28e8ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e26d452393e4256b18196c585186f24"
I20260927 13:50:05.193082 28088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5c9d5cc66749adb47485bb2be0e90e" candidate_uuid: "be53d36199114b9fbb19b95c28e8ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43"
I20260927 13:50:05.193277 28377 raft_consensus.cc:3037] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:05.193286 28088 raft_consensus.cc:3037] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:05.195565 28377 raft_consensus.cc:2454] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be53d36199114b9fbb19b95c28e8ee48 in term 1.
I20260927 13:50:05.196012 28088 raft_consensus.cc:2454] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be53d36199114b9fbb19b95c28e8ee48 in term 1.
I20260927 13:50:05.196043 28170 leader_election.cc:302] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e26d452393e4256b18196c585186f24, be53d36199114b9fbb19b95c28e8ee48; no voters:
I20260927 13:50:05.196388 28458 raft_consensus.cc:2781] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:05.196933 28458 raft_consensus.cc:686] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 1 LEADER]: Becoming Leader. State: Replica: be53d36199114b9fbb19b95c28e8ee48, State: Running, Role: LEADER
I20260927 13:50:05.197577 28458 consensus_queue.cc:237] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } }
I20260927 13:50:05.202459 27156 catalog_manager.cc:5495] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to be53d36199114b9fbb19b95c28e8ee48 (127.20.133.66). New cstate: current_term: 1 leader_uuid: "be53d36199114b9fbb19b95c28e8ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } health_report { overall_health: UNKNOWN } } }
I20260927 13:50:05.223016 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:05.217738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.20.133.66:46135)
0927 13:50:05.217907 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:05.217912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:05.217929 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:50:05.218021 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:05.218024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:05.218114 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:05.218245 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:50:05.218537 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.219274 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.220704 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.221179 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.221298 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.221862 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:50:05.222402 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:50:05.222517 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:05.222715 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 13:50:05.222796 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:50:05.231544 28377 raft_consensus.cc:1261] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24 [term 1 FOLLOWER]: Refusing update from remote peer be53d36199114b9fbb19b95c28e8ee48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:05.231544 28088 raft_consensus.cc:1261] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 1 FOLLOWER]: Refusing update from remote peer be53d36199114b9fbb19b95c28e8ee48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:05.232576 28465 consensus_queue.cc:1035] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:05.232962 28458 consensus_queue.cc:1035] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 13:50:05.240072 28284 tablet.cc:2369] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:50:05.272302 28471 mvcc.cc:205] Tried to move back new op lower bound from 7333957653417529344 to 7333957653292294144. Current Snapshot: MvccSnapshot[applied={T|T < 7333957653417529344}]
I20260927 13:50:05.282949 27156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:05.283227 27156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:05.290563 27156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25826f702bd249f280d27611d1ce1ad2: Sending DeleteTablet for 3 replicas of tablet bb5c9d5cc66749adb47485bb2be0e90e
I20260927 13:50:05.291810 28068 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5c9d5cc66749adb47485bb2be0e90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:05 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:45110
I20260927 13:50:05.291806 28216 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5c9d5cc66749adb47485bb2be0e90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:05 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:58774
I20260927 13:50:05.292814 28479 tablet_replica.cc:330] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: stopping tablet replica
I20260927 13:50:05.292894 28357 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5c9d5cc66749adb47485bb2be0e90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:05 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:35598
I20260927 13:50:05.293286 28479 raft_consensus.cc:2229] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:05.293807 28479 raft_consensus.cc:2258] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:05.302987 28479 ts_tablet_manager.cc:1905] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:05.305656 28480 tablet_replica.cc:330] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: stopping tablet replica
I20260927 13:50:05.306087 28480 raft_consensus.cc:2229] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:50:05.306742 28480 raft_consensus.cc:2258] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:05.306959 28481 tablet_replica.cc:330] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: stopping tablet replica
I20260927 13:50:05.307408 28481 raft_consensus.cc:2229] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:05.307957 28481 raft_consensus.cc:2258] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:05.314430 28480 ts_tablet_manager.cc:1905] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:05.315464 28479 ts_tablet_manager.cc:1918] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:05.315665 28479 log.cc:1196] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/bb5c9d5cc66749adb47485bb2be0e90e
I20260927 13:50:05.316254 28479 ts_tablet_manager.cc:1939] T bb5c9d5cc66749adb47485bb2be0e90e P 26bd1ea161b74c5aab11c5a5d20a2e43: Deleting consensus metadata
I20260927 13:50:05.318555 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 27125
I20260927 13:50:05.319511 28481 ts_tablet_manager.cc:1905] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:05.319655 27187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:05.322713 28480 ts_tablet_manager.cc:1918] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:05.322922 28480 log.cc:1196] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/bb5c9d5cc66749adb47485bb2be0e90e
I20260927 13:50:05.323597 28480 ts_tablet_manager.cc:1939] T bb5c9d5cc66749adb47485bb2be0e90e P be53d36199114b9fbb19b95c28e8ee48: Deleting consensus metadata
I20260927 13:50:05.332840 28481 ts_tablet_manager.cc:1918] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:05.333998 28481 log.cc:1196] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/bb5c9d5cc66749adb47485bb2be0e90e
I20260927 13:50:05.336758 28481 ts_tablet_manager.cc:1939] T bb5c9d5cc66749adb47485bb2be0e90e P 9e26d452393e4256b18196c585186f24: Deleting consensus metadata
I20260927 13:50:05.341737 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:05.296804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.20.133.125:33637)
0927 13:50:05.296977 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:05.296984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:05.317133 (+ 20149us) server_negotiation.cc:407] Connection header received
0927 13:50:05.320674 (+ 3541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:05.320680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:05.320783 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:05.320857 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:05.325631 (+ 4774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.326874 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.328806 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.329437 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.329564 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.333052 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:05.333059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:05.333063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:05.333116 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:05.336473 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:05.339427 (+ 2954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:05.339434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:05.339437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:05.339604 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:05.340037 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:05.340042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:05.340045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:05.340471 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:05.340766 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:05.341101 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 13:50:05.341355 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 13:50:05.357980 28431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:05.317480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52404 (local address 127.20.133.124:38229)
0927 13:50:05.317661 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:05.317666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:05.341778 (+ 24112us) server_negotiation.cc:407] Connection header received
0927 13:50:05.342038 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:05.342045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:05.342134 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:05.342220 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:05.344142 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.345308 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.347218 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.347905 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.348052 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.351342 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:05.351350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:05.351355 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:05.351401 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:05.354927 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:05.355734 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:05.355740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:05.355744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:05.355896 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:05.356312 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:05.356318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:05.356321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:05.356737 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:05.357006 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:05.357374 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 13:50:05.357609 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:50:05.453229 27142 catalog_manager.cc:4841] TS 9e26d452393e4256b18196c585186f24 (127.20.133.67:44205): tablet bb5c9d5cc66749adb47485bb2be0e90e (table test-table [id=7be8418332024e04abe0901f1fbfe0bc]) successfully deleted
I20260927 13:50:05.457026 27144 catalog_manager.cc:4841] TS 26bd1ea161b74c5aab11c5a5d20a2e43 (127.20.133.65:38349): tablet bb5c9d5cc66749adb47485bb2be0e90e (table test-table [id=7be8418332024e04abe0901f1fbfe0bc]) successfully deleted
I20260927 13:50:05.457828 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 27125
I20260927 13:50:05.457791 27142 catalog_manager.cc:4841] TS be53d36199114b9fbb19b95c28e8ee48 (127.20.133.66:46135): tablet bb5c9d5cc66749adb47485bb2be0e90e (table test-table [id=7be8418332024e04abe0901f1fbfe0bc]) successfully deleted
I20260927 13:50:05.497704 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:05.481547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.20.133.125:33637)
0927 13:50:05.481755 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:05.481761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:05.482477 (+ 716us) server_negotiation.cc:407] Connection header received
0927 13:50:05.482554 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:05.482560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:05.482646 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:05.482762 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:05.484568 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.485297 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.486913 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.487363 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.487469 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.491243 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:05.491248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:05.491252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:05.491285 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:05.494452 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:05.495283 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:05.495289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:05.495293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:05.495444 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:05.495897 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:05.495903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:05.495907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:05.496207 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:05.496585 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:05.497114 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 13:50:05.497366 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:50:05.498538 28431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:05.482219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.20.133.124:38229)
0927 13:50:05.482429 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:05.482435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:05.482993 (+ 558us) server_negotiation.cc:407] Connection header received
0927 13:50:05.483141 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:05.483146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:05.483225 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:05.483340 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:05.485294 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.486369 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.488187 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:05.488809 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:05.488906 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:05.492218 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:05.492224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:05.492229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:05.492271 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:05.495406 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:05.496360 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:05.496366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:05.496369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:05.496510 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:05.496907 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:05.496912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:05.496915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:05.497277 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:05.497500 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:05.498012 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 13:50:05.498242 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:50:06.261159 28490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.260143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.65:40891)
0927 13:50:06.260747 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.260926 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":301,"threads_started":1}
W20260927 13:50:06.261631 28134 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:33783 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
I20260927 13:50:06.275992 28491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.275089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.66:54813)
0927 13:50:06.275633 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.275781 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":255,"threads_started":1}
W20260927 13:50:06.276476 28282 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:33783 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
I20260927 13:50:06.279554 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.278694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.67:57413)
0927 13:50:06.279173 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.279335 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":229,"threads_started":1}
W20260927 13:50:06.280025 28423 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:33783 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
I20260927 13:50:06.790666 28493 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25826f702bd249f280d27611d1ce1ad2)
I20260927 13:50:06.790807 28494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25826f702bd249f280d27611d1ce1ad2)
I20260927 13:50:06.791054 28494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } }
I20260927 13:50:06.790989 28493 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } }
I20260927 13:50:06.791981 28494 leader_election.cc:288] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783), cd7597d7a00a4223aabb7168756d8ebd (127.20.133.124:38229)
I20260927 13:50:06.802804 28493 leader_election.cc:288] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783), 858b03dc507a47ea9997ce955053284a (127.20.133.125:33637)
I20260927 13:50:06.803606 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.792779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44450)
0927 13:50:06.802864 (+ 10085us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.803157 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9860,"threads_started":1}
I20260927 13:50:06.805015 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.792325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44434)
0927 13:50:06.804403 (+ 12078us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.804677 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11882,"threads_started":1}
W20260927 13:50:06.805588 27747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
W20260927 13:50:06.805588 27937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
I20260927 13:50:06.808446 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.807769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44466)
0927 13:50:06.808051 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.808171 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260927 13:50:06.808859 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.808037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44482)
0927 13:50:06.808249 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.808675 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260927 13:50:06.808866 27937 leader_election.cc:334] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783): Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
W20260927 13:50:06.809109 27747 leader_election.cc:334] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783): Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
I20260927 13:50:06.813616 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.792474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:38229 (local address 127.0.0.1:52414)
0927 13:50:06.804761 (+ 12287us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.804782 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:06.804950 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:06.806160 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:06.806166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:06.806198 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:06.806944 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:06.806965 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:06.809247 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:06.809252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:06.811107 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:06.811127 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:06.812695 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:06.812700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:06.812869 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:06.812972 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 13:50:06.813070 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 13:50:06.813209 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12136,"mutex_wait_us":51,"thread_start_us":11972,"threads_started":1}
I20260927 13:50:06.813699 28500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.804170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.20.133.124:38229)
0927 13:50:06.805639 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:06.805646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:06.805673 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:50:06.805881 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:06.805889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:06.805979 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:06.806418 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:06.807561 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:06.809077 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:06.811332 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:06.812481 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:06.812674 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:06.812756 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:06.813131 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 13:50:06.813288 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1322,"thread_start_us":1175,"threads_started":1}
I20260927 13:50:06.814530 27964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "858b03dc507a47ea9997ce955053284a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cd7597d7a00a4223aabb7168756d8ebd" is_pre_election: true
I20260927 13:50:06.815040 27964 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 858b03dc507a47ea9997ce955053284a in term 1.
I20260927 13:50:06.815150 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.797065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:33637 (local address 127.0.0.1:54706)
0927 13:50:06.803910 (+ 6845us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.803936 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:06.804142 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:06.808071 (+ 3929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:06.808077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:06.808118 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:06.808724 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:06.808743 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:06.810484 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:06.810491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:06.812397 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:06.812414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:06.814222 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:06.814228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:06.814451 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:06.814486 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:50:06.814589 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 13:50:06.814736 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6542,"mutex_wait_us":70,"thread_start_us":5356,"threads_started":1}
I20260927 13:50:06.815209 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.797118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.20.133.125:33637)
0927 13:50:06.806988 (+ 9870us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:06.806995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:06.807025 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:50:06.807121 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:06.807128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:06.807226 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:06.808940 (+ 1714us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:06.809268 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:06.810333 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:06.812611 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:06.813987 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:06.814161 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:06.814232 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:06.814693 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 13:50:06.814844 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":9679,"thread_start_us":9422,"threads_started":1}
I20260927 13:50:06.815845 27748 leader_election.cc:302] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 858b03dc507a47ea9997ce955053284a, cd7597d7a00a4223aabb7168756d8ebd; no voters: 25826f702bd249f280d27611d1ce1ad2
I20260927 13:50:06.815927 27773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd7597d7a00a4223aabb7168756d8ebd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "858b03dc507a47ea9997ce955053284a" is_pre_election: true
I20260927 13:50:06.816258 27773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7597d7a00a4223aabb7168756d8ebd in term 1.
I20260927 13:50:06.816460 28494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 13:50:06.816598 28494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 25826f702bd249f280d27611d1ce1ad2)
I20260927 13:50:06.816722 28494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:50:06.816974 27939 leader_election.cc:302] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 858b03dc507a47ea9997ce955053284a, cd7597d7a00a4223aabb7168756d8ebd; no voters: 25826f702bd249f280d27611d1ce1ad2
I20260927 13:50:06.817514 28493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 13:50:06.817692 28493 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 25826f702bd249f280d27611d1ce1ad2)
I20260927 13:50:06.817790 28493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:50:06.819538 28494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } }
I20260927 13:50:06.820279 28494 leader_election.cc:288] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 2 election: Requested vote from peers 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783), cd7597d7a00a4223aabb7168756d8ebd (127.20.133.124:38229)
I20260927 13:50:06.820142 28493 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } }
I20260927 13:50:06.820973 27964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "858b03dc507a47ea9997ce955053284a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cd7597d7a00a4223aabb7168756d8ebd"
I20260927 13:50:06.821102 28493 leader_election.cc:288] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [CANDIDATE]: Term 2 election: Requested vote from peers 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783), 858b03dc507a47ea9997ce955053284a (127.20.133.125:33637)
I20260927 13:50:06.821302 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.820736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44488)
0927 13:50:06.820964 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.821089 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 13:50:06.821346 27964 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 858b03dc507a47ea9997ce955053284a in current term 2: Already voted for candidate cd7597d7a00a4223aabb7168756d8ebd in this term.
I20260927 13:50:06.821735 27773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd7597d7a00a4223aabb7168756d8ebd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "858b03dc507a47ea9997ce955053284a"
I20260927 13:50:06.822074 27773 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cd7597d7a00a4223aabb7168756d8ebd in current term 2: Already voted for candidate 858b03dc507a47ea9997ce955053284a in this term.
I20260927 13:50:06.822513 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.821942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44492)
0927 13:50:06.822192 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.822315 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 13:50:06.823181 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.822806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44498)
0927 13:50:06.822965 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.823050 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 13:50:06.823596 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:06.823189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44502)
0927 13:50:06.823370 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:06.823442 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 13:50:06.823778 27937 leader_election.cc:334] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783): Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
W20260927 13:50:06.823803 27747 leader_election.cc:334] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783): Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
I20260927 13:50:06.823899 27937 leader_election.cc:302] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd7597d7a00a4223aabb7168756d8ebd; no voters: 25826f702bd249f280d27611d1ce1ad2, 858b03dc507a47ea9997ce955053284a
I20260927 13:50:06.823909 27747 leader_election.cc:302] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 858b03dc507a47ea9997ce955053284a; no voters: 25826f702bd249f280d27611d1ce1ad2, cd7597d7a00a4223aabb7168756d8ebd
I20260927 13:50:06.824196 28493 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 13:50:06.824240 28494 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 13:50:07.263774 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:07.262561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.65:40991)
0927 13:50:07.263300 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:07.263456 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":391,"threads_started":1}
I20260927 13:50:07.278359 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:07.277348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.66:60069)
0927 13:50:07.278038 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:07.278166 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":379,"threads_started":1}
I20260927 13:50:07.281759 28505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:07.280913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.67:58609)
0927 13:50:07.281433 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:07.281559 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":261,"threads_started":1}
I20260927 13:50:08.265926 28506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.264897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.65:37969)
0927 13:50:08.265515 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.265686 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":294,"threads_started":1}
I20260927 13:50:08.280541 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.279530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.66:53997)
0927 13:50:08.280106 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.280224 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":290,"threads_started":1}
I20260927 13:50:08.283742 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.282790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.67:39059)
0927 13:50:08.283385 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.283498 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":305,"threads_started":1}
I20260927 13:50:08.810101 28509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:50:08.810330 28509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } }
I20260927 13:50:08.811000 28509 leader_election.cc:288] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783), cd7597d7a00a4223aabb7168756d8ebd (127.20.133.124:38229)
I20260927 13:50:08.811859 27964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "858b03dc507a47ea9997ce955053284a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cd7597d7a00a4223aabb7168756d8ebd" is_pre_election: true
I20260927 13:50:08.812240 27964 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 858b03dc507a47ea9997ce955053284a in term 2.
I20260927 13:50:08.812311 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.811248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44804)
0927 13:50:08.811840 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.812038 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":277,"threads_started":1}
I20260927 13:50:08.812850 27748 leader_election.cc:302] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 858b03dc507a47ea9997ce955053284a, cd7597d7a00a4223aabb7168756d8ebd; no voters:
I20260927 13:50:08.813174 28509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 13:50:08.813297 28509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:50:08.813395 28509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 2 FOLLOWER]: Advancing to term 3
I20260927 13:50:08.813913 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.813507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44820)
0927 13:50:08.813682 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.813773 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 13:50:08.814141 27747 leader_election.cc:334] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783): Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
I20260927 13:50:08.815912 28509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } }
I20260927 13:50:08.816584 28509 leader_election.cc:288] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 3 election: Requested vote from peers 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783), cd7597d7a00a4223aabb7168756d8ebd (127.20.133.124:38229)
I20260927 13:50:08.817193 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.816757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44826)
0927 13:50:08.816947 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.817047 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 13:50:08.817253 27964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "858b03dc507a47ea9997ce955053284a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cd7597d7a00a4223aabb7168756d8ebd"
I20260927 13:50:08.817528 27964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 2 FOLLOWER]: Advancing to term 3
I20260927 13:50:08.818084 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.817787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44836)
0927 13:50:08.817922 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.817988 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 13:50:08.818230 27747 leader_election.cc:334] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783): Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
I20260927 13:50:08.819757 27964 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 858b03dc507a47ea9997ce955053284a in term 3.
I20260927 13:50:08.820212 27748 leader_election.cc:302] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 858b03dc507a47ea9997ce955053284a, cd7597d7a00a4223aabb7168756d8ebd; no voters: 25826f702bd249f280d27611d1ce1ad2
I20260927 13:50:08.820603 28509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 3 FOLLOWER]: Leader election won for term 3
I20260927 13:50:08.821053 28509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [term 3 LEADER]: Becoming Leader. State: Replica: 858b03dc507a47ea9997ce955053284a, State: Running, Role: LEADER
I20260927 13:50:08.821511 28509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } }
I20260927 13:50:08.823233 28512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 858b03dc507a47ea9997ce955053284a. Latest consensus state: current_term: 3 leader_uuid: "858b03dc507a47ea9997ce955053284a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:08.823578 28512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:08.824324 28514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:50:08.827834 28514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:50:08.828702 28514 catalog_manager.cc:1259] Loaded cluster ID: a47906cfaa1d4b80bc3aac7116ef0a20
I20260927 13:50:08.828778 28514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:50:08.829994 28514 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:50:08.830770 28514 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: Loaded TSK: 0
I20260927 13:50:08.831557 28514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:50:08.872951 27763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:50:08.879985 27763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:50:08.885063 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.884577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44846)
0927 13:50:08.884772 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.884864 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 13:50:08.885097 27964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [term 3 FOLLOWER]: Refusing update from remote peer 858b03dc507a47ea9997ce955053284a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 13:50:08.885946 28512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 13:50:08.886365 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.885903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44850)
0927 13:50:08.886088 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.886185 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 13:50:08.886582 27747 consensus_peers.cc:487] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a -> Peer 25826f702bd249f280d27611d1ce1ad2 (127.20.133.126:33783): Couldn't send request to peer 25826f702bd249f280d27611d1ce1ad2. Status: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 13:50:08.889437 28516 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 858b03dc507a47ea9997ce955053284a. Latest consensus state: current_term: 3 leader_uuid: "858b03dc507a47ea9997ce955053284a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:08.889838 28516 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:50:08.890982 28512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "858b03dc507a47ea9997ce955053284a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:08.891338 28512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:08.892155 28516 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "858b03dc507a47ea9997ce955053284a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "25826f702bd249f280d27611d1ce1ad2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33783 } } peers { permanent_uuid: "858b03dc507a47ea9997ce955053284a" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "cd7597d7a00a4223aabb7168756d8ebd" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 38229 } attrs { promote: false } } }
I20260927 13:50:08.895565 28516 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7597d7a00a4223aabb7168756d8ebd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:50:08.909988 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.901486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:44205 (local address 127.0.0.1:35610)
0927 13:50:08.901803 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.901824 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:08.901952 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:08.903055 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:08.903061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:08.903098 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:08.903449 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:08.903460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.905003 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.905010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:08.907122 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:08.907142 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.908987 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.908995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:08.909245 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:08.909297 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 13:50:08.909381 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 13:50:08.909566 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260927 13:50:08.910967 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.901663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.20.133.67:44205)
0927 13:50:08.902689 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:08.902695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:08.902715 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:50:08.902791 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:08.902795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:08.902868 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:08.903014 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:08.903583 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.904770 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.907419 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.908733 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.910270 (+ 1537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:08.910386 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:08.910450 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 13:50:08.910635 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":764,"threads_started":1}
I20260927 13:50:08.911814 28357 tablet_service.cc:1467] Processing CreateTablet for tablet 36bc4b0f3cb64eed9f567fde20e15e06 (DEFAULT_TABLE table=test-table [id=8b8f76a7ac304bf882017890ee4069dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:08.912364 28357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bc4b0f3cb64eed9f567fde20e15e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:08.916586 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.902947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.20.133.66:46135)
0927 13:50:08.908936 (+ 5989us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:08.908945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:08.909733 (+ 788us) server_negotiation.cc:407] Connection header received
0927 13:50:08.909840 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:08.909845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:08.909943 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:08.910051 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:08.910860 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.911876 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.914577 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.915467 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.915603 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:08.915729 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:08.916070 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 13:50:08.916231 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5790,"thread_start_us":5623,"threads_started":1}
I20260927 13:50:08.916651 28525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.902577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:46135 (local address 127.0.0.1:58792)
0927 13:50:08.909440 (+ 6863us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.909461 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:08.909620 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:08.910256 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:08.910260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:08.910286 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:08.910683 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:08.910694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.912001 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.912007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:08.914204 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:08.914217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.915714 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.915720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:08.915890 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:08.915923 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:50:08.916025 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 13:50:08.916188 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6730,"mutex_wait_us":3,"thread_start_us":6582,"threads_started":1}
I20260927 13:50:08.917752 28216 tablet_service.cc:1467] Processing CreateTablet for tablet 36bc4b0f3cb64eed9f567fde20e15e06 (DEFAULT_TABLE table=test-table [id=8b8f76a7ac304bf882017890ee4069dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:08.918341 28216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bc4b0f3cb64eed9f567fde20e15e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:08.919359 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.901940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38349 (local address 127.0.0.1:45126)
0927 13:50:08.903040 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.903060 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:08.903177 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:08.911660 (+ 8483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:08.911665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:08.911708 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:08.912117 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:08.912132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.914379 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.914385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:08.916515 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:08.916531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.918373 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.918381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:08.918585 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:08.918643 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 13:50:08.918719 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 13:50:08.918887 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":33,"thread_start_us":802,"threads_started":1}
I20260927 13:50:08.920835 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.902200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.20.133.65:38349)
0927 13:50:08.911253 (+ 9053us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:08.911262 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:08.911284 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:50:08.911378 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:08.911384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:08.911472 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:08.911583 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:08.912417 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.914223 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.916796 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:08.918083 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:08.920113 (+ 2030us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:08.920241 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:08.920302 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 13:50:08.920488 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8892,"thread_start_us":8765,"threads_started":1}
I20260927 13:50:08.921650 28068 tablet_service.cc:1467] Processing CreateTablet for tablet 36bc4b0f3cb64eed9f567fde20e15e06 (DEFAULT_TABLE table=test-table [id=8b8f76a7ac304bf882017890ee4069dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:08.922164 28068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bc4b0f3cb64eed9f567fde20e15e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:08.924103 28528 tablet_bootstrap.cc:492] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48: Bootstrap starting.
I20260927 13:50:08.926112 28527 tablet_bootstrap.cc:492] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24: Bootstrap starting.
I20260927 13:50:08.926950 28528 tablet_bootstrap.cc:654] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:08.928202 28527 tablet_bootstrap.cc:654] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:08.928889 28530 tablet_bootstrap.cc:492] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43: Bootstrap starting.
I20260927 13:50:08.929977 28528 tablet_bootstrap.cc:492] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48: No bootstrap required, opened a new log
I20260927 13:50:08.930160 28528 ts_tablet_manager.cc:1397] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 13:50:08.931002 28530 tablet_bootstrap.cc:654] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:08.931120 28527 tablet_bootstrap.cc:492] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24: No bootstrap required, opened a new log
I20260927 13:50:08.931038 28528 raft_consensus.cc:348] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } }
I20260927 13:50:08.931290 28528 raft_consensus.cc:374] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:08.931384 28528 raft_consensus.cc:729] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be53d36199114b9fbb19b95c28e8ee48, State: Initialized, Role: FOLLOWER
I20260927 13:50:08.931622 28528 consensus_queue.cc:260] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } }
I20260927 13:50:08.931804 28527 ts_tablet_manager.cc:1397] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 13:50:08.932639 28528 ts_tablet_manager.cc:1428] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 13:50:08.932816 28527 raft_consensus.cc:348] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } }
I20260927 13:50:08.933084 28527 raft_consensus.cc:374] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:08.933184 28527 raft_consensus.cc:729] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e26d452393e4256b18196c585186f24, State: Initialized, Role: FOLLOWER
I20260927 13:50:08.933422 28527 consensus_queue.cc:260] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } }
I20260927 13:50:08.936088 28283 heartbeater.cc:503] Master 127.20.133.125:33637 was elected leader, sending a full tablet report...
I20260927 13:50:08.941313 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.933364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.66:41363)
0927 13:50:08.940896 (+ 7532us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.941059 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7369,"thread_start_us":7238,"threads_started":1}
I20260927 13:50:08.942344 28527 ts_tablet_manager.cc:1428] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 13:50:08.942781 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.942187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.66:59287)
0927 13:50:08.942486 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.942600 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260927 13:50:08.944526 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.942639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.67:56879)
0927 13:50:08.943369 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.943522 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":614,"threads_started":1}
I20260927 13:50:08.945211 28530 tablet_bootstrap.cc:492] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43: No bootstrap required, opened a new log
I20260927 13:50:08.945420 28530 ts_tablet_manager.cc:1397] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20260927 13:50:08.945643 28424 heartbeater.cc:503] Master 127.20.133.125:33637 was elected leader, sending a full tablet report...
I20260927 13:50:08.946416 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.945240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.67:59399)
0927 13:50:08.945864 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.946073 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"threads_started":1}
I20260927 13:50:08.946489 28530 raft_consensus.cc:348] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } }
I20260927 13:50:08.946770 28530 raft_consensus.cc:374] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:08.946895 28530 raft_consensus.cc:729] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26bd1ea161b74c5aab11c5a5d20a2e43, State: Initialized, Role: FOLLOWER
I20260927 13:50:08.947227 28530 consensus_queue.cc:260] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } }
I20260927 13:50:08.948400 28530 ts_tablet_manager.cc:1428] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 13:50:08.949919 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.948901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.65:35861)
0927 13:50:08.949516 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.949671 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":327,"threads_started":1}
I20260927 13:50:08.950907 28135 heartbeater.cc:503] Master 127.20.133.125:33637 was elected leader, sending a full tablet report...
I20260927 13:50:08.952134 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:08.950744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.65:59147)
0927 13:50:08.951830 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:08.951947 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260927 13:50:09.192976 28535 raft_consensus.cc:482] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:50:09.193317 28535 raft_consensus.cc:504] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } }
I20260927 13:50:09.195369 28535 leader_election.cc:288] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26bd1ea161b74c5aab11c5a5d20a2e43 (127.20.133.65:38349), be53d36199114b9fbb19b95c28e8ee48 (127.20.133.66:46135)
I20260927 13:50:09.203354 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.195964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:47743 (local address 127.20.133.65:38349)
0927 13:50:09.196241 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:09.196248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:09.196436 (+ 188us) server_negotiation.cc:407] Connection header received
0927 13:50:09.196697 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:09.196704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:09.196782 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:09.196861 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:09.197353 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.198381 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.200952 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.202143 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.202319 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:09.202400 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:09.202898 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 13:50:09.203033 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 13:50:09.203346 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.195876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:33887 (local address 127.20.133.66:46135)
0927 13:50:09.196125 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:09.196130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:09.196152 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:50:09.196328 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:09.196358 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:09.196436 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:09.196540 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:09.197105 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.198158 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.200972 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.202162 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.202319 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:09.202400 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:09.202898 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 13:50:09.203033 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 13:50:09.203447 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.195764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38349 (local address 127.20.133.67:47743)
0927 13:50:09.196305 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:09.196321 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:09.196511 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:09.196926 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:09.196930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:09.196954 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:09.197220 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:09.197228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.198517 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.198521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:09.200775 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:09.200788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.202400 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.202407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:09.202690 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:09.202762 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 13:50:09.202862 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 13:50:09.203062 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":366}
I20260927 13:50:09.203646 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.195746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:46135 (local address 127.20.133.67:33887)
0927 13:50:09.195967 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:09.195989 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:09.196167 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:09.196598 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:09.196603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:09.196629 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:09.196936 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:09.196948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.198301 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.198305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:09.200771 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:09.200788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.202400 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.202407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:09.202688 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:09.202757 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 13:50:09.202862 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 13:50:09.203254 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":129}
I20260927 13:50:09.204243 28088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bc4b0f3cb64eed9f567fde20e15e06" candidate_uuid: "9e26d452393e4256b18196c585186f24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" is_pre_election: true
I20260927 13:50:09.204614 28088 raft_consensus.cc:2454] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e26d452393e4256b18196c585186f24 in term 0.
I20260927 13:50:09.204833 28236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bc4b0f3cb64eed9f567fde20e15e06" candidate_uuid: "9e26d452393e4256b18196c585186f24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be53d36199114b9fbb19b95c28e8ee48" is_pre_election: true
I20260927 13:50:09.205319 28236 raft_consensus.cc:2454] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e26d452393e4256b18196c585186f24 in term 0.
I20260927 13:50:09.205291 28313 leader_election.cc:302] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26bd1ea161b74c5aab11c5a5d20a2e43, 9e26d452393e4256b18196c585186f24; no voters:
I20260927 13:50:09.205821 28535 raft_consensus.cc:2781] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:50:09.205950 28535 raft_consensus.cc:482] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:50:09.206074 28535 raft_consensus.cc:3037] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:09.208258 28535 raft_consensus.cc:504] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } }
I20260927 13:50:09.209005 28535 leader_election.cc:288] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [CANDIDATE]: Term 1 election: Requested vote from peers 26bd1ea161b74c5aab11c5a5d20a2e43 (127.20.133.65:38349), be53d36199114b9fbb19b95c28e8ee48 (127.20.133.66:46135)
I20260927 13:50:09.209712 28088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bc4b0f3cb64eed9f567fde20e15e06" candidate_uuid: "9e26d452393e4256b18196c585186f24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43"
I20260927 13:50:09.209939 28088 raft_consensus.cc:3037] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:09.211987 28088 raft_consensus.cc:2454] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e26d452393e4256b18196c585186f24 in term 1.
I20260927 13:50:09.212548 28313 leader_election.cc:302] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26bd1ea161b74c5aab11c5a5d20a2e43, 9e26d452393e4256b18196c585186f24; no voters:
I20260927 13:50:09.212494 28236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bc4b0f3cb64eed9f567fde20e15e06" candidate_uuid: "9e26d452393e4256b18196c585186f24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be53d36199114b9fbb19b95c28e8ee48"
I20260927 13:50:09.212738 28236 raft_consensus.cc:3037] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:09.212978 28535 raft_consensus.cc:2781] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:09.213579 28535 raft_consensus.cc:686] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 1 LEADER]: Becoming Leader. State: Replica: 9e26d452393e4256b18196c585186f24, State: Running, Role: LEADER
I20260927 13:50:09.214542 28535 consensus_queue.cc:237] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } }
I20260927 13:50:09.214962 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.214361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.67:40245)
0927 13:50:09.214586 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:09.214735 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 13:50:09.215473 28236 raft_consensus.cc:2454] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e26d452393e4256b18196c585186f24 in term 1.
I20260927 13:50:09.219421 27763 catalog_manager.cc:5495] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e26d452393e4256b18196c585186f24 (127.20.133.67). New cstate: current_term: 1 leader_uuid: "9e26d452393e4256b18196c585186f24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e26d452393e4256b18196c585186f24" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 } health_report { overall_health: UNKNOWN } } }
I20260927 13:50:09.290868 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.285032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.20.133.67:44205)
0927 13:50:09.285271 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:09.285276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:09.285299 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:50:09.285550 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:09.285554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:09.285667 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:09.285820 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:50:09.286139 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.286938 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.288177 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.288706 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.288809 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:09.289360 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:50:09.290208 (+ 848us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:50:09.290346 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:09.290454 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 13:50:09.290539 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 13:50:09.295954 28088 raft_consensus.cc:1261] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 1 FOLLOWER]: Refusing update from remote peer 9e26d452393e4256b18196c585186f24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:09.296518 28236 raft_consensus.cc:1261] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48 [term 1 FOLLOWER]: Refusing update from remote peer 9e26d452393e4256b18196c585186f24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:09.296736 28540 consensus_queue.cc:1035] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26bd1ea161b74c5aab11c5a5d20a2e43" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:09.297562 28535 consensus_queue.cc:1035] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be53d36199114b9fbb19b95c28e8ee48" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:09.300714 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.299644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.65:47629)
0927 13:50:09.300236 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:09.300435 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419}
I20260927 13:50:09.304102 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.303521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.66:51335)
0927 13:50:09.303717 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:09.303844 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 13:50:09.309042 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.308209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.65:53435)
0927 13:50:09.308693 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:09.308821 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346}
I20260927 13:50:09.315300 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.314694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.67:58109)
0927 13:50:09.314953 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:09.315094 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 13:50:09.315879 28546 mvcc.cc:205] Tried to move back new op lower bound from 7333957670067314688 to 7333957669747965952. Current Snapshot: MvccSnapshot[applied={T|T < 7333957670067314688 or (T in {7333957670067314688})}]
I20260927 13:50:09.317147 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.316628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.20.133.66:40635)
0927 13:50:09.316812 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:09.316918 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 13:50:09.329618 27761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:09.330065 27761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:09.337853 27761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 858b03dc507a47ea9997ce955053284a: Sending DeleteTablet for 3 replicas of tablet 36bc4b0f3cb64eed9f567fde20e15e06
I20260927 13:50:09.339310 28357 tablet_service.cc:1514] Processing DeleteTablet for tablet 36bc4b0f3cb64eed9f567fde20e15e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:09 UTC) from {username='oryx', principal='oryx/127.20.133.125@KRBTEST.COM'} at 127.0.0.1:35610
I20260927 13:50:09.339555 28068 tablet_service.cc:1514] Processing DeleteTablet for tablet 36bc4b0f3cb64eed9f567fde20e15e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:09 UTC) from {username='oryx', principal='oryx/127.20.133.125@KRBTEST.COM'} at 127.0.0.1:45126
I20260927 13:50:09.339789 28216 tablet_service.cc:1514] Processing DeleteTablet for tablet 36bc4b0f3cb64eed9f567fde20e15e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:09 UTC) from {username='oryx', principal='oryx/127.20.133.125@KRBTEST.COM'} at 127.0.0.1:58792
I20260927 13:50:09.345084 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 27941
I20260927 13:50:09.352917 28555 tablet_replica.cc:330] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48: stopping tablet replica
I20260927 13:50:09.353229 28553 tablet_replica.cc:330] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24: stopping tablet replica
I20260927 13:50:09.353426 28553 raft_consensus.cc:2229] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:50:09.354030 28555 raft_consensus.cc:2229] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:09.354216 28553 raft_consensus.cc:2258] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:09.354514 28555 raft_consensus.cc:2258] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:09.355201 28554 tablet_replica.cc:330] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43: stopping tablet replica
I20260927 13:50:09.355983 28553 ts_tablet_manager.cc:1905] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:09.356397 28555 ts_tablet_manager.cc:1905] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:09.356671 28554 raft_consensus.cc:2229] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:09.357941 28554 raft_consensus.cc:2258] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:09.364589 28554 ts_tablet_manager.cc:1905] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:09.364866 28555 ts_tablet_manager.cc:1918] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:09.365070 28555 log.cc:1196] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/36bc4b0f3cb64eed9f567fde20e15e06
I20260927 13:50:09.365615 28555 ts_tablet_manager.cc:1939] T 36bc4b0f3cb64eed9f567fde20e15e06 P be53d36199114b9fbb19b95c28e8ee48: Deleting consensus metadata
I20260927 13:50:09.366981 27748 catalog_manager.cc:4841] TS be53d36199114b9fbb19b95c28e8ee48 (127.20.133.66:46135): tablet 36bc4b0f3cb64eed9f567fde20e15e06 (table test-table [id=8b8f76a7ac304bf882017890ee4069dc]) successfully deleted
I20260927 13:50:09.367689 28553 ts_tablet_manager.cc:1918] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:09.367929 28553 log.cc:1196] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/36bc4b0f3cb64eed9f567fde20e15e06
I20260927 13:50:09.368819 28553 ts_tablet_manager.cc:1939] T 36bc4b0f3cb64eed9f567fde20e15e06 P 9e26d452393e4256b18196c585186f24: Deleting consensus metadata
I20260927 13:50:09.370363 27748 catalog_manager.cc:4841] TS 9e26d452393e4256b18196c585186f24 (127.20.133.67:44205): tablet 36bc4b0f3cb64eed9f567fde20e15e06 (table test-table [id=8b8f76a7ac304bf882017890ee4069dc]) successfully deleted
I20260927 13:50:09.372741 28554 ts_tablet_manager.cc:1918] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:09.372910 28554 log.cc:1196] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/36bc4b0f3cb64eed9f567fde20e15e06
I20260927 13:50:09.373505 28554 ts_tablet_manager.cc:1939] T 36bc4b0f3cb64eed9f567fde20e15e06 P 26bd1ea161b74c5aab11c5a5d20a2e43: Deleting consensus metadata
I20260927 13:50:09.375133 27750 catalog_manager.cc:4841] TS 26bd1ea161b74c5aab11c5a5d20a2e43 (127.20.133.65:38349): tablet 36bc4b0f3cb64eed9f567fde20e15e06 (table test-table [id=8b8f76a7ac304bf882017890ee4069dc]) successfully deleted
I20260927 13:50:09.379113 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.347171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.20.133.65:38349)
0927 13:50:09.347398 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:09.347405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:09.359298 (+ 11893us) server_negotiation.cc:407] Connection header received
0927 13:50:09.359613 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:09.359619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:09.359748 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:09.359866 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:09.362274 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.363518 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.365668 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.366459 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.366584 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:09.371038 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:09.371074 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:09.371096 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:09.371154 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:09.375715 (+ 4561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:09.376844 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:09.376853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:09.376855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:09.377046 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:09.377507 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:09.377511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:09.377513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:09.377930 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:09.378173 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:09.378591 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 13:50:09.378787 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 13:50:09.379982 28128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:09.395620 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.394967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44866)
0927 13:50:09.395215 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:09.395384 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 13:50:09.397300 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.396822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:33783 (local address 127.0.0.1:44868)
0927 13:50:09.396999 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:09.397114 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:33783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 13:50:09.486192 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 27941
I20260927 13:50:09.501374 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 28145
I20260927 13:50:09.518554 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.502206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.20.133.66:46135)
0927 13:50:09.502383 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:09.502388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:09.502572 (+ 184us) server_negotiation.cc:407] Connection header received
0927 13:50:09.502803 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:09.502809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:09.502893 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:09.503000 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:09.505073 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.505891 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.507844 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.508296 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.508625 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:09.512229 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:09.512248 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:09.512264 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:09.512327 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:09.515284 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:09.516216 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:09.516225 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:09.516228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:09.516479 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:09.516966 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:09.516970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:09.516972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:09.517414 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:09.517802 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:09.518047 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 13:50:09.518252 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:50:09.519255 28276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:09.641108 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 28145
I20260927 13:50:09.672986 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 28294
I20260927 13:50:09.687773 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:09.673906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.20.133.67:44205)
0927 13:50:09.674119 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:09.674125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:09.674261 (+ 136us) server_negotiation.cc:407] Connection header received
0927 13:50:09.674475 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:09.674479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:09.674580 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:09.674668 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:09.676024 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.677159 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.678376 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:09.678833 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:09.678951 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:09.681533 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:09.681550 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:09.681569 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:09.681610 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:09.684859 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:09.685590 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:09.685597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:09.685600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:09.685783 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:09.686178 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:09.686183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:09.686186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:09.686596 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:09.686943 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:09.687191 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 13:50:09.687411 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 13:50:09.688808 28417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:09.819679 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 28294
I20260927 13:50:09.840142 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 27618
I20260927 13:50:09.841334 27793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:09.966199 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 27618
W20260927 13:50:09.984032 27937 connection.cc:535] server connection from 127.0.0.1:52414 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 13:50:09.984463 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 27718
I20260927 13:50:09.985744 27984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:10.104115 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 27718
2026-09-27T13:50:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11987 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:50:10 dist-test-slave-q25r krb5kdc[28566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:50:10 dist-test-slave-q25r krb5kdc[28566](info): set up 2 sockets
Sep 27 13:50:10 dist-test-slave-q25r krb5kdc[28566](info): commencing operation
krb5kdc: starting...
W20260927 13:50:11.137810 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:50:11 dist-test-slave-q25r krb5kdc[28566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:50:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.126.keytab.
Entry for principal oryx/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.126.keytab.
I20260927 13:50:11.397205 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:35867
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.126.keytab
--principal=oryx/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:35867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:11.780279 28579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:11.780717 28579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:11.780867 28579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:11.788887 28579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:50:11.789047 28579 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 13:50:11.789091 28579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:11.789184 28579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:11.789258 28579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:50:11.789342 28579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:50:11.800625 28579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:35867
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.126.keytab
--principal=oryx/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:35867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:11.802402 28579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:11.804039 28579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:11.818288 28584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:11.818416 28585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:11.818416 28587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:11.819428 28579 server_base.cc:1034] running on GCE node
I20260927 13:50:11.820492 28579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:11.822290 28579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:11.823575 28579 hybrid_clock.cc:648] HybridClock initialized: now 1790517011823482 us; error 61 us; skew 500 ppm
I20260927 13:50:11.823972 28579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:11 dist-test-slave-q25r krb5kdc[28566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517011, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:11.827433 28579 init.cc:377] Logged in from keytab as oryx/127.20.133.126@KRBTEST.COM (short username oryx)
I20260927 13:50:11.829200 28579 webserver.cc:460] Webserver started at http://127.20.133.126:41937/ using document root <none> and password file <none>
I20260927 13:50:11.830011 28579 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:11.830132 28579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:11.830467 28579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:11.833868 28579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "a3457716c0554f75b221da3536bee34e"
format_stamp: "Formatted at 2026-09-27 13:50:11 on dist-test-slave-q25r"
server_key: "3eea3162518425d61966bd19c407f0c0"
server_key_iv: "38d8e92c97419cb2f0114ea663d32c51"
server_key_version: "encryptionkey@0"
I20260927 13:50:11.834714 28579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "a3457716c0554f75b221da3536bee34e"
format_stamp: "Formatted at 2026-09-27 13:50:11 on dist-test-slave-q25r"
server_key: "3eea3162518425d61966bd19c407f0c0"
server_key_iv: "38d8e92c97419cb2f0114ea663d32c51"
server_key_version: "encryptionkey@0"
I20260927 13:50:11.841372 28579 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 13:50:11.845829 28594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:11.847589 28579 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260927 13:50:11.847841 28579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "a3457716c0554f75b221da3536bee34e"
format_stamp: "Formatted at 2026-09-27 13:50:11 on dist-test-slave-q25r"
server_key: "3eea3162518425d61966bd19c407f0c0"
server_key_iv: "38d8e92c97419cb2f0114ea663d32c51"
server_key_version: "encryptionkey@0"
I20260927 13:50:11.848085 28579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:11.869328 28579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:11.874418 28579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:11.874779 28579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:11.898346 28579 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:35867
I20260927 13:50:11.898598 28655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:35867 every 8 connection(s)
I20260927 13:50:11.900107 28579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:50:11.903625 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 28579
I20260927 13:50:11.903834 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:50:11.904615 28656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:11.904691 21013 external_mini_cluster.cc:1428] Setting key 14c01b487bae0ffc334c9733ee2ddaea
I20260927 13:50:11.919660 28656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e: Bootstrap starting.
Sep 27 13:50:11 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:50:11.923970 28656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:11.925408 28656 log.cc:824] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:11.928637 28656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e: No bootstrap required, opened a new log
I20260927 13:50:11.931589 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:11.907003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.20.133.126:35867)
0927 13:50:11.907937 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:11.907952 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:11.908007 (+ 55us) server_negotiation.cc:407] Connection header received
0927 13:50:11.909895 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:11.909923 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:11.910439 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:11.910876 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:11.912748 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:11.914248 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:11.916187 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:11.917174 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:11.917345 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:11.922505 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:11.922516 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:11.922538 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:11.922600 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:11.926561 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:11.927441 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:11.927450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:11.927458 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:11.927689 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:11.928098 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:11.928102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:11.928104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:11.928869 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:11.929202 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:11.929683 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 13:50:11.930205 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":419,"threads_started":1}
I20260927 13:50:11.937888 28656 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3457716c0554f75b221da3536bee34e" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35867 } }
I20260927 13:50:11.938309 28656 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:11.938403 28656 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3457716c0554f75b221da3536bee34e, State: Initialized, Role: FOLLOWER
I20260927 13:50:11.938993 28656 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3457716c0554f75b221da3536bee34e" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35867 } }
I20260927 13:50:11.939220 28656 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:50:11.939344 28656 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:50:11.939498 28656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:11.941699 28656 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3457716c0554f75b221da3536bee34e" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35867 } }
I20260927 13:50:11.942206 28656 leader_election.cc:302] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3457716c0554f75b221da3536bee34e; no voters:
I20260927 13:50:11.942833 28656 leader_election.cc:288] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:50:11.943076 28661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:11.944363 28661 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [term 1 LEADER]: Becoming Leader. State: Replica: a3457716c0554f75b221da3536bee34e, State: Running, Role: LEADER
I20260927 13:50:11.945058 28661 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3457716c0554f75b221da3536bee34e" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35867 } }
I20260927 13:50:11.945230 28656 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:50:11.948596 28663 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3457716c0554f75b221da3536bee34e. Latest consensus state: current_term: 1 leader_uuid: "a3457716c0554f75b221da3536bee34e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3457716c0554f75b221da3536bee34e" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35867 } } }
I20260927 13:50:11.949100 28662 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3457716c0554f75b221da3536bee34e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3457716c0554f75b221da3536bee34e" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35867 } } }
I20260927 13:50:11.949410 28663 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:11.949509 28662 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:11.950456 28668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:50:11.957199 28668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:50:11.969112 28668 catalog_manager.cc:1347] Generated new cluster ID: c3b0e2e8d0cf4c45934a046bda91f7d0
I20260927 13:50:11.969260 28668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:50:11.985652 28668 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:50:11.987180 28668 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:50:12.000500 28668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e: Generated new TSK 0
I20260927 13:50:12.002148 28668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab.
Entry for principal oryx/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab.
I20260927 13:50:12.065969 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:35867
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab
--principal=oryx/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:12.433228 28682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:12.433558 28682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:12.433723 28682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:12.441601 28682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:12.441738 28682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:12.441852 28682 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:50:12.441951 28682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:50:12.453147 28682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab
--principal=oryx/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.133.126:35867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:12.454902 28682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:12.456445 28682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:12.469947 28690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:12.470027 28688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:12.469925 28687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:12.470913 28682 server_base.cc:1034] running on GCE node
I20260927 13:50:12.471560 28682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:12.472576 28682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:12.473824 28682 hybrid_clock.cc:648] HybridClock initialized: now 1790517012473734 us; error 58 us; skew 500 ppm
I20260927 13:50:12.474139 28682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:12 dist-test-slave-q25r krb5kdc[28566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517012, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:12.477324 28682 init.cc:377] Logged in from keytab as oryx/127.20.133.65@KRBTEST.COM (short username oryx)
I20260927 13:50:12.478837 28682 webserver.cc:460] Webserver started at http://127.20.133.65:43473/ using document root <none> and password file <none>
I20260927 13:50:12.479540 28682 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:12.479635 28682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:12.479898 28682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:12.482963 28682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "fa030a9b1082446f8e6856dfaa240000"
format_stamp: "Formatted at 2026-09-27 13:50:12 on dist-test-slave-q25r"
server_key: "50555332a91351cb6b84554e9e2e0855"
server_key_iv: "c39dd805030b23d031fd3148efb772bc"
server_key_version: "encryptionkey@0"
I20260927 13:50:12.483817 28682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "fa030a9b1082446f8e6856dfaa240000"
format_stamp: "Formatted at 2026-09-27 13:50:12 on dist-test-slave-q25r"
server_key: "50555332a91351cb6b84554e9e2e0855"
server_key_iv: "c39dd805030b23d031fd3148efb772bc"
server_key_version: "encryptionkey@0"
I20260927 13:50:12.489858 28682 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260927 13:50:12.493726 28697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:12.495240 28682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:50:12.495404 28682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "fa030a9b1082446f8e6856dfaa240000"
format_stamp: "Formatted at 2026-09-27 13:50:12 on dist-test-slave-q25r"
server_key: "50555332a91351cb6b84554e9e2e0855"
server_key_iv: "c39dd805030b23d031fd3148efb772bc"
server_key_version: "encryptionkey@0"
I20260927 13:50:12.495611 28682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:12.518707 28682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:12.523041 28682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:12.523309 28682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:12.526065 28682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:12.526154 28682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:12.526276 28682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:12.526330 28682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:50:12 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517012, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:50:12.551401 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:12.526651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:44957 (local address 127.20.133.126:35867)
0927 13:50:12.527248 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:12.527255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:12.528431 (+ 1176us) server_negotiation.cc:407] Connection header received
0927 13:50:12.530627 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:12.530633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:12.530746 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:12.530865 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:12.533621 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:12.534528 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:12.537593 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:12.538200 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:12.538345 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:12.543719 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:12.543728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:12.543734 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:12.543794 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:12.547350 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:12.548440 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:12.548446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:12.548450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:12.548617 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:12.549219 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:12.549224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:12.549228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:12.549539 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:12.549700 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:12.550699 (+ 999us) server_negotiation.cc:299] Negotiation successful
0927 13:50:12.550985 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":329,"threads_started":1}
I20260927 13:50:12.552379 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:12.527324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:35867 (local address 127.20.133.65:44957)
0927 13:50:12.528226 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:12.528276 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:12.530299 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:12.531205 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:12.531221 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:12.532244 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:12.533351 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:12.533388 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:12.534770 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:12.534777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:12.537408 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:12.537431 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:12.538502 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:12.538511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:12.538882 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:12.539810 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:12.539852 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:12.543380 (+ 3528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:12.547530 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:12.547544 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:12.547570 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:12.548285 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:12.548748 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:12.548754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:12.548757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:12.549076 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:12.549787 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:12.549800 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:12.550255 (+ 455us) client_negotiation.cc:769] Sending connection context
0927 13:50:12.550518 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 13:50:12.550970 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":352,"threads_started":1}
I20260927 13:50:12.574554 28682 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:36575
I20260927 13:50:12.574663 28817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:36575 every 8 connection(s)
I20260927 13:50:12.576310 28682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:50:12.577167 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 28682
I20260927 13:50:12.577399 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:50:12.577567 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:12.559379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:45365 (local address 127.20.133.126:35867)
0927 13:50:12.559586 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:12.559593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:12.560197 (+ 604us) server_negotiation.cc:407] Connection header received
0927 13:50:12.560486 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:12.560493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:12.560621 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:12.560806 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:12.562384 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:12.563126 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:12.566279 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:12.567118 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:12.567251 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:12.570364 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:12.570372 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:12.570377 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:12.570427 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:12.573305 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:12.574817 (+ 1512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:12.574823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:12.574827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:12.574996 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:12.575489 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:12.575494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:12.575497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:12.575902 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:12.576068 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:12.576865 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 13:50:12.577166 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:50:12.577744 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:12.559355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:35867 (local address 127.20.133.65:45365)
0927 13:50:12.560049 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:12.560072 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:12.560256 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:12.560888 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:12.560896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:12.561671 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:12.562150 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:12.562167 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:12.563795 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:12.563803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:12.565946 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:12.565977 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:12.567677 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:12.567686 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:12.567927 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:12.568894 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:12.568917 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:12.570107 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:12.573704 (+ 3597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:12.573726 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:12.573732 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:12.574591 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:12.575159 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:12.575164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:12.575167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:12.575348 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:12.576098 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:12.576103 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:12.576474 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 13:50:12.576704 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 13:50:12.577280 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":350,"threads_started":1}
I20260927 13:50:12.578392 21013 external_mini_cluster.cc:1428] Setting key 7a7f791883397be141ae7f64b404227f
I20260927 13:50:12.581080 28818 heartbeater.cc:347] Connected to a master server at 127.20.133.126:35867
I20260927 13:50:12.581496 28818 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:12.582587 28818 heartbeater.cc:511] Master 127.20.133.126:35867 requested a full tablet report, sending...
I20260927 13:50:12.585175 28611 ts_manager.cc:194] Registered new tserver with Master: fa030a9b1082446f8e6856dfaa240000 (127.20.133.65:36575)
I20260927 13:50:12.587180 28611 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:44957
WARNING: no policy specified for oryx/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab.
Entry for principal oryx/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab.
I20260927 13:50:12.629662 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:35867
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab
--principal=oryx/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:12.974606 28822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:12.974917 28822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:12.975046 28822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:12.982573 28822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:12.982728 28822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:12.982856 28822 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:50:12.982969 28822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:50:12.993453 28822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab
--principal=oryx/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.133.126:35867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:12.994899 28822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:12.996250 28822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:13.010102 28828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:13.010026 28827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:13.009997 28830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:13.010779 28822 server_base.cc:1034] running on GCE node
I20260927 13:50:13.011559 28822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:13.012701 28822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:13.013962 28822 hybrid_clock.cc:648] HybridClock initialized: now 1790517013013882 us; error 53 us; skew 500 ppm
I20260927 13:50:13.014304 28822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:13 dist-test-slave-q25r krb5kdc[28566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517013, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:13.017827 28822 init.cc:377] Logged in from keytab as oryx/127.20.133.66@KRBTEST.COM (short username oryx)
I20260927 13:50:13.019454 28822 webserver.cc:460] Webserver started at http://127.20.133.66:40437/ using document root <none> and password file <none>
I20260927 13:50:13.020252 28822 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:13.020408 28822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:13.020725 28822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:13.023952 28822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "337c0ed1b3e648a691bb711be0a84b82"
format_stamp: "Formatted at 2026-09-27 13:50:13 on dist-test-slave-q25r"
server_key: "544f5b28d631417818a36a30dfab9685"
server_key_iv: "2d1729645a8d4e8f3c7ddcb6d53e0409"
server_key_version: "encryptionkey@0"
I20260927 13:50:13.024870 28822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "337c0ed1b3e648a691bb711be0a84b82"
format_stamp: "Formatted at 2026-09-27 13:50:13 on dist-test-slave-q25r"
server_key: "544f5b28d631417818a36a30dfab9685"
server_key_iv: "2d1729645a8d4e8f3c7ddcb6d53e0409"
server_key_version: "encryptionkey@0"
I20260927 13:50:13.030571 28822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:50:13.034499 28837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:13.035944 28822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 13:50:13.036109 28822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "337c0ed1b3e648a691bb711be0a84b82"
format_stamp: "Formatted at 2026-09-27 13:50:13 on dist-test-slave-q25r"
server_key: "544f5b28d631417818a36a30dfab9685"
server_key_iv: "2d1729645a8d4e8f3c7ddcb6d53e0409"
server_key_version: "encryptionkey@0"
I20260927 13:50:13.036310 28822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:13.051321 28822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:13.056268 28822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:13.056565 28822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:13.059250 28822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:13.059358 28822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:13.059484 28822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:13.059542 28822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:50:13 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517013, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:50:13.087118 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.060631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:35867 (local address 127.20.133.66:39625)
0927 13:50:13.061522 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:13.061570 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:13.063593 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:13.064771 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:13.064780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:13.065696 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:13.066684 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.066710 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.068132 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.068139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.071042 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.071068 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.071966 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.071975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.072382 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.073272 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:13.073298 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:13.076571 (+ 3273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:13.080992 (+ 4421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:13.081009 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:13.081038 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:13.081823 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:13.082322 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:13.082329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:13.082332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:13.082772 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:13.083685 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:13.083693 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:13.084173 (+ 480us) client_negotiation.cc:769] Sending connection context
0927 13:50:13.084643 (+ 470us) client_negotiation.cc:240] Negotiation successful
0927 13:50:13.085370 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":339,"threads_started":1}
I20260927 13:50:13.088729 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.060021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:39625 (local address 127.20.133.126:35867)
0927 13:50:13.060203 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:13.060209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:13.061712 (+ 1503us) server_negotiation.cc:407] Connection header received
0927 13:50:13.063979 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:13.063984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:13.064081 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:13.064163 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:13.066898 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.067879 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.071261 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.071716 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.071845 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.076888 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:13.076896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:13.076900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:13.076948 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:13.080744 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:13.082025 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:13.082031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:13.082035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:13.082172 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:13.082934 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:13.082941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:13.082945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:13.083375 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:13.083599 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:13.088067 (+ 4468us) server_negotiation.cc:299] Negotiation successful
0927 13:50:13.088307 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:50:13.107240 28822 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:40613
I20260927 13:50:13.107255 28956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:40613 every 8 connection(s)
I20260927 13:50:13.108846 28822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:50:13.110991 28957 heartbeater.cc:347] Connected to a master server at 127.20.133.126:35867
I20260927 13:50:13.111253 28957 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:13.111267 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.095388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:37117 (local address 127.20.133.126:35867)
0927 13:50:13.095600 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:13.095607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:13.096150 (+ 543us) server_negotiation.cc:407] Connection header received
0927 13:50:13.096316 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:13.096323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:13.096438 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:13.096551 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:13.097952 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.099210 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.101648 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.102389 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.102545 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.104680 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:13.104686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:13.104689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:13.104738 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:13.108034 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:13.108849 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:13.108855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:13.108858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:13.109033 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:13.109457 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:13.109462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:13.109465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:13.109957 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:13.110121 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:13.110652 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 13:50:13.110876 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:50:13.111371 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.095382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:35867 (local address 127.20.133.66:37117)
0927 13:50:13.096045 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:13.096065 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:13.096202 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:13.096634 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:13.096641 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:13.097312 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:13.097737 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.097754 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.099414 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.099420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.101419 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.101445 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.102674 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.102681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.102810 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.103601 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:13.103622 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:13.104444 (+ 822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:13.108211 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:13.108216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:13.108219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:13.108711 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:13.109161 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:13.109165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:13.109167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:13.109311 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:13.110189 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:13.110195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:13.110382 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 13:50:13.110504 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 13:50:13.110871 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":346,"threads_started":1}
I20260927 13:50:13.112182 28957 heartbeater.cc:511] Master 127.20.133.126:35867 requested a full tablet report, sending...
I20260927 13:50:13.112811 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 28822
I20260927 13:50:13.112963 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:50:13.113379 21013 external_mini_cluster.cc:1428] Setting key 7e657102fc1b6b523289401af581bcaf
I20260927 13:50:13.114665 28610 ts_manager.cc:194] Registered new tserver with Master: 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613)
I20260927 13:50:13.115670 28610 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:39625
WARNING: no policy specified for oryx/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab.
Entry for principal oryx/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab.
I20260927 13:50:13.162143 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:35867
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab
--principal=oryx/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:13.495700 28961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:13.496021 28961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:13.496157 28961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:13.503862 28961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:13.504016 28961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:13.504135 28961 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:50:13.504245 28961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:50:13.514421 28961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab
--principal=oryx/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.133.126:35867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:13.515758 28961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:13.517104 28961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:13.530622 28966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:13.530730 28967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:13.531579 28969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:13.532052 28961 server_base.cc:1034] running on GCE node
I20260927 13:50:13.532747 28961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:13.533733 28961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:13.534969 28961 hybrid_clock.cc:648] HybridClock initialized: now 1790517013534890 us; error 51 us; skew 500 ppm
I20260927 13:50:13.535290 28961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:13 dist-test-slave-q25r krb5kdc[28566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517013, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:13.538390 28961 init.cc:377] Logged in from keytab as oryx/127.20.133.67@KRBTEST.COM (short username oryx)
I20260927 13:50:13.540016 28961 webserver.cc:460] Webserver started at http://127.20.133.67:40525/ using document root <none> and password file <none>
I20260927 13:50:13.540810 28961 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:13.540902 28961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:13.541203 28961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:13.544193 28961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "07c837dfccdf4dadad4ce7026d001677"
format_stamp: "Formatted at 2026-09-27 13:50:13 on dist-test-slave-q25r"
server_key: "067ae6f964de81a2676dde610004ac9a"
server_key_iv: "9b77ff3a20a402d04088c8be1338a245"
server_key_version: "encryptionkey@0"
I20260927 13:50:13.545054 28961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "07c837dfccdf4dadad4ce7026d001677"
format_stamp: "Formatted at 2026-09-27 13:50:13 on dist-test-slave-q25r"
server_key: "067ae6f964de81a2676dde610004ac9a"
server_key_iv: "9b77ff3a20a402d04088c8be1338a245"
server_key_version: "encryptionkey@0"
I20260927 13:50:13.550637 28961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:50:13.554913 28976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:13.556905 28961 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:50:13.557145 28961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "07c837dfccdf4dadad4ce7026d001677"
format_stamp: "Formatted at 2026-09-27 13:50:13 on dist-test-slave-q25r"
server_key: "067ae6f964de81a2676dde610004ac9a"
server_key_iv: "9b77ff3a20a402d04088c8be1338a245"
server_key_version: "encryptionkey@0"
I20260927 13:50:13.557395 28961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:13.581704 28961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:13.586086 28961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:13.586355 28961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:13.589701 28961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:13.589807 28961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:13.589931 28961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:13.589987 28961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:13.591292 28818 heartbeater.cc:503] Master 127.20.133.126:35867 was elected leader, sending a full tablet report...
Sep 27 13:50:13 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517013, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:50:13.620700 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.590621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:45913 (local address 127.20.133.126:35867)
0927 13:50:13.590878 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:13.590885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:13.593315 (+ 2430us) server_negotiation.cc:407] Connection header received
0927 13:50:13.597041 (+ 3726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:13.597050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:13.597175 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:13.597326 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:13.600690 (+ 3364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.601685 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.604742 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.605383 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.605511 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.610999 (+ 5488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:13.611005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:13.611009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:13.611054 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:13.614804 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:13.615879 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:13.615884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:13.615887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:13.616072 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:13.616688 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:13.616693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:13.616696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:13.617082 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:13.617265 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:13.620111 (+ 2846us) server_negotiation.cc:299] Negotiation successful
0927 13:50:13.620412 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 13:50:13.621253 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.591918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:35867 (local address 127.20.133.67:45913)
0927 13:50:13.593081 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:13.593157 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:13.596543 (+ 3386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:13.597820 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:13.597839 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:13.598969 (+ 1130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:13.600324 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.600402 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.601921 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.601929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.604562 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.604582 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.605661 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.605669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.606079 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.607075 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:13.607117 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:13.610673 (+ 3556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:13.615071 (+ 4398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:13.615084 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:13.615113 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:13.615711 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:13.616190 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:13.616194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:13.616197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:13.616508 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:13.617357 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:13.617367 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:13.618018 (+ 651us) client_negotiation.cc:769] Sending connection context
0927 13:50:13.618350 (+ 332us) client_negotiation.cc:240] Negotiation successful
0927 13:50:13.619122 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":468,"threads_started":1}
I20260927 13:50:13.648435 28961 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:44015
I20260927 13:50:13.648504 29095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:44015 every 8 connection(s)
I20260927 13:50:13.650128 28961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:50:13.652817 29096 heartbeater.cc:347] Connected to a master server at 127.20.133.126:35867
I20260927 13:50:13.653065 29096 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:13.653824 29096 heartbeater.cc:511] Master 127.20.133.126:35867 requested a full tablet report, sending...
I20260927 13:50:13.653918 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.636204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:57671 (local address 127.20.133.126:35867)
0927 13:50:13.636492 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:13.636500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:13.637093 (+ 593us) server_negotiation.cc:407] Connection header received
0927 13:50:13.637399 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:13.637405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:13.637512 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:13.637638 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:13.639170 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.640064 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.642657 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.643139 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.643255 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.646286 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:13.646294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:13.646299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:13.646348 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:13.649914 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:13.651100 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:13.651107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:13.651111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:13.651322 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:13.652026 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:13.652032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:13.652035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:13.652514 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:13.652714 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:13.653219 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 13:50:13.653515 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 13:50:13.654112 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.636029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:35867 (local address 127.20.133.67:57671)
0927 13:50:13.636916 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:13.636945 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:13.637167 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:13.637710 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:13.637716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:13.638421 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:13.638850 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.638865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.640261 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.640268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.642442 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.642465 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.643375 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.643382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.643590 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.644419 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:13.644445 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:13.645898 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:13.650120 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:13.650126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:13.650130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:13.650865 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:13.651602 (+ 737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:13.651607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:13.651611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:13.651837 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:13.652656 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:13.652660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:13.652919 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 13:50:13.653079 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 13:50:13.653515 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":446,"threads_started":1}
I20260927 13:50:13.655442 28611 ts_manager.cc:194] Registered new tserver with Master: 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015)
I20260927 13:50:13.656471 28611 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:45913
I20260927 13:50:13.656745 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 28961
I20260927 13:50:13.656931 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:50:13.657478 21013 external_mini_cluster.cc:1428] Setting key 2c50ccd34ef4ab884d47f44b2a2e86b0
I20260927 13:50:13.661157 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:50:13 dist-test-slave-q25r krb5kdc[28566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517013, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:50:13 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:13 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:50:13.719393 28706 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:13.708062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.20.133.126:35867)
0927 13:50:13.708380 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:13.708387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:13.708568 (+ 181us) server_negotiation.cc:407] Connection header received
0927 13:50:13.708775 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:13.708780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:13.708854 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:13.708941 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:13.710325 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.711087 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.712325 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.712791 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.712907 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.719052 (+ 6145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52558: BlockingRecv error: recv got EOF from 127.0.0.1:52558 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 27 13:50:13 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517013, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:50:13.739056 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.723536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.20.133.126:35867)
0927 13:50:13.723789 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:13.723796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:13.724076 (+ 280us) server_negotiation.cc:407] Connection header received
0927 13:50:13.724245 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:13.724250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:13.724412 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:13.724517 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:13.726038 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.726793 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.728178 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.728678 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.728785 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.733478 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:13.733484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:13.733488 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:13.733536 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:13.736213 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:13.737203 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:13.737207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:13.737210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:13.737343 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:13.737720 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:13.737723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:13.737725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:13.737987 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:13.738280 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:13.738601 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 13:50:13.738808 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 13:50:13.743918 28611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:50:13.751456 28611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:50:13.780758 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.766966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36575 (local address 127.0.0.1:58750)
0927 13:50:13.768080 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:13.768148 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:13.768448 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:13.770172 (+ 1724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:13.770182 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:13.770221 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:13.770782 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.770803 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.773184 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.773192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.776208 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.776242 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.779333 (+ 3091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.779342 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.779671 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.779735 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 13:50:13.779938 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 13:50:13.780236 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":584,"threads_started":1}
I20260927 13:50:13.782938 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.767499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.20.133.65:36575)
0927 13:50:13.769053 (+ 1554us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:13.769067 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:13.769108 (+ 41us) server_negotiation.cc:407] Connection header received
0927 13:50:13.769242 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:13.769253 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:13.769596 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:13.769843 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:13.771060 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.772918 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.776583 (+ 3665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.778567 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.778757 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.779091 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:13.782253 (+ 3162us) server_negotiation.cc:299] Negotiation successful
0927 13:50:13.782522 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1312,"thread_start_us":638,"threads_started":1}
I20260927 13:50:13.793409 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.768001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:44015 (local address 127.0.0.1:50972)
0927 13:50:13.781136 (+ 13135us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:13.781161 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:13.781299 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:13.784766 (+ 3467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:13.784772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:13.784801 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:13.785131 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.785148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.787939 (+ 2791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.787945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.790067 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.790088 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.792558 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.792565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.792749 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.792792 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 13:50:13.792868 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 13:50:13.793034 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12985,"thread_start_us":13580,"threads_started":1}
I20260927 13:50:13.794638 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.768543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:40613 (local address 127.0.0.1:38558)
0927 13:50:13.781896 (+ 13353us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:13.781915 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:13.782038 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:13.786329 (+ 4291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:13.786334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:13.786360 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:13.786811 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.786834 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.789251 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.789257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.791376 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:13.791393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.793903 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.793910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:13.794072 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.794103 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 13:50:13.794168 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 13:50:13.794314 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13205,"thread_start_us":20438,"threads_started":1}
I20260927 13:50:13.794361 28748 tablet_service.cc:1467] Processing CreateTablet for tablet fc7b3dcbe5f2441085f5caf1ab4fafd0 (DEFAULT_TABLE table=test-table [id=c4344d2cfba74f6d8acd90991a7b1534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:13.795151 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.768447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.20.133.67:44015)
0927 13:50:13.783922 (+ 15475us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:13.783933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:13.783976 (+ 43us) server_negotiation.cc:407] Connection header received
0927 13:50:13.784103 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:13.784113 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:13.784509 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:13.784760 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:13.785478 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.787755 (+ 2277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.790353 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.792372 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.793962 (+ 1590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.794252 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:13.794340 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 13:50:13.794541 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15243,"thread_start_us":15026,"threads_started":1}
I20260927 13:50:13.795742 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:13.769104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38558 (local address 127.20.133.66:40613)
0927 13:50:13.785605 (+ 16501us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:13.785617 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:13.785650 (+ 33us) server_negotiation.cc:407] Connection header received
0927 13:50:13.785766 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:13.785775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:13.786098 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:13.787024 (+ 926us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:13.787247 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.788713 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.792255 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:13.793650 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:13.794941 (+ 1291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:13.795169 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:13.795257 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 13:50:13.795448 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16298,"thread_start_us":16144,"threads_started":1}
I20260927 13:50:13.796271 28748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7b3dcbe5f2441085f5caf1ab4fafd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:13.800724 28887 tablet_service.cc:1467] Processing CreateTablet for tablet fc7b3dcbe5f2441085f5caf1ab4fafd0 (DEFAULT_TABLE table=test-table [id=c4344d2cfba74f6d8acd90991a7b1534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:13.801236 29026 tablet_service.cc:1467] Processing CreateTablet for tablet fc7b3dcbe5f2441085f5caf1ab4fafd0 (DEFAULT_TABLE table=test-table [id=c4344d2cfba74f6d8acd90991a7b1534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:13.802311 28887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7b3dcbe5f2441085f5caf1ab4fafd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:13.802706 29026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7b3dcbe5f2441085f5caf1ab4fafd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:13.818121 29121 tablet_bootstrap.cc:492] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: Bootstrap starting.
I20260927 13:50:13.820076 29120 tablet_bootstrap.cc:492] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: Bootstrap starting.
I20260927 13:50:13.823022 29121 tablet_bootstrap.cc:654] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:13.824440 29120 tablet_bootstrap.cc:654] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:13.824658 29122 tablet_bootstrap.cc:492] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: Bootstrap starting.
I20260927 13:50:13.824980 29121 log.cc:824] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:13.825803 29120 log.cc:824] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:13.829609 29121 tablet_bootstrap.cc:492] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: No bootstrap required, opened a new log
I20260927 13:50:13.830137 29121 ts_tablet_manager.cc:1397] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260927 13:50:13.830390 29122 tablet_bootstrap.cc:654] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:13.830643 29120 tablet_bootstrap.cc:492] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: No bootstrap required, opened a new log
I20260927 13:50:13.830989 29120 ts_tablet_manager.cc:1397] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260927 13:50:13.831771 29122 log.cc:824] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:13.841213 29122 tablet_bootstrap.cc:492] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: No bootstrap required, opened a new log
I20260927 13:50:13.841589 29122 ts_tablet_manager.cc:1397] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s
I20260927 13:50:13.842386 29120 raft_consensus.cc:348] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:13.842883 29120 raft_consensus.cc:374] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:13.842990 29120 raft_consensus.cc:729] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa030a9b1082446f8e6856dfaa240000, State: Initialized, Role: FOLLOWER
I20260927 13:50:13.843713 29120 consensus_queue.cc:260] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:13.845809 29120 ts_tablet_manager.cc:1428] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
I20260927 13:50:13.844604 29121 raft_consensus.cc:348] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:13.846665 29121 raft_consensus.cc:374] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:13.846781 29121 raft_consensus.cc:729] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c837dfccdf4dadad4ce7026d001677, State: Initialized, Role: FOLLOWER
I20260927 13:50:13.847517 29121 consensus_queue.cc:260] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:13.849083 29096 heartbeater.cc:503] Master 127.20.133.126:35867 was elected leader, sending a full tablet report...
I20260927 13:50:13.849776 29121 ts_tablet_manager.cc:1428] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: Time spent starting tablet: real 0.019s user 0.010s sys 0.006s
I20260927 13:50:13.854391 29122 raft_consensus.cc:348] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:13.855027 29122 raft_consensus.cc:374] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:13.855154 29122 raft_consensus.cc:729] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337c0ed1b3e648a691bb711be0a84b82, State: Initialized, Role: FOLLOWER
I20260927 13:50:13.856005 29122 consensus_queue.cc:260] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:13.857336 28957 heartbeater.cc:503] Master 127.20.133.126:35867 was elected leader, sending a full tablet report...
I20260927 13:50:13.857908 29122 ts_tablet_manager.cc:1428] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
W20260927 13:50:13.860997 28958 tablet.cc:2369] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:50:13.902424 29097 tablet.cc:2369] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:50:14.051491 29127 raft_consensus.cc:482] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:50:14.051899 29127 raft_consensus.cc:504] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.053761 29127 leader_election.cc:288] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa030a9b1082446f8e6856dfaa240000 (127.20.133.65:36575), 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613)
I20260927 13:50:14.061375 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.054361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:44563 (local address 127.20.133.65:36575)
0927 13:50:14.054642 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:14.054649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:14.054744 (+ 95us) server_negotiation.cc:407] Connection header received
0927 13:50:14.055055 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:14.055062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:14.055171 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:14.055274 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:14.055814 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.056899 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.059112 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.060247 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.060450 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.060524 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:14.060902 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 13:50:14.061044 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 13:50:14.061419 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.054334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36575 (local address 127.20.133.67:44563)
0927 13:50:14.054581 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:14.054607 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:14.054859 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:14.055357 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:14.055363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:14.055403 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:14.055621 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.055631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.057056 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.057062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.058907 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.058924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.060452 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.060458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.060694 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.060739 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 13:50:14.060872 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 13:50:14.061055 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":115}
I20260927 13:50:14.062125 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.054977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:35191 (local address 127.20.133.66:40613)
0927 13:50:14.055216 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:14.055222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:14.055511 (+ 289us) server_negotiation.cc:407] Connection header received
0927 13:50:14.055772 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:14.055778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:14.055856 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:14.055933 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:14.056694 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.057757 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.059935 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.061007 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.061169 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.061246 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:14.061696 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 13:50:14.061891 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 13:50:14.062703 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.054761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:40613 (local address 127.20.133.67:35191)
0927 13:50:14.055400 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:14.055418 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:14.055612 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:14.055985 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:14.055990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:14.056016 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:14.056484 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.056499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.057941 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.057946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.059714 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.059732 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.061272 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.061278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.061470 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.061504 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:50:14.062192 (+ 688us) client_negotiation.cc:240] Negotiation successful
0927 13:50:14.062361 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":8,"thread_start_us":360,"threads_started":1}
I20260927 13:50:14.062732 28768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7b3dcbe5f2441085f5caf1ab4fafd0" candidate_uuid: "07c837dfccdf4dadad4ce7026d001677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa030a9b1082446f8e6856dfaa240000" is_pre_election: true
I20260927 13:50:14.063304 28768 raft_consensus.cc:2454] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c837dfccdf4dadad4ce7026d001677 in term 0.
I20260927 13:50:14.063777 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7b3dcbe5f2441085f5caf1ab4fafd0" candidate_uuid: "07c837dfccdf4dadad4ce7026d001677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337c0ed1b3e648a691bb711be0a84b82" is_pre_election: true
I20260927 13:50:14.064062 28978 leader_election.cc:302] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c837dfccdf4dadad4ce7026d001677, fa030a9b1082446f8e6856dfaa240000; no voters:
I20260927 13:50:14.064502 28907 raft_consensus.cc:2454] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c837dfccdf4dadad4ce7026d001677 in term 0.
I20260927 13:50:14.064616 29127 raft_consensus.cc:2781] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:50:14.064760 29127 raft_consensus.cc:482] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:50:14.064859 29127 raft_consensus.cc:3037] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:14.067195 29127 raft_consensus.cc:504] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.067906 29127 leader_election.cc:288] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [CANDIDATE]: Term 1 election: Requested vote from peers fa030a9b1082446f8e6856dfaa240000 (127.20.133.65:36575), 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613)
I20260927 13:50:14.068606 28768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7b3dcbe5f2441085f5caf1ab4fafd0" candidate_uuid: "07c837dfccdf4dadad4ce7026d001677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa030a9b1082446f8e6856dfaa240000"
I20260927 13:50:14.068636 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7b3dcbe5f2441085f5caf1ab4fafd0" candidate_uuid: "07c837dfccdf4dadad4ce7026d001677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337c0ed1b3e648a691bb711be0a84b82"
I20260927 13:50:14.068805 28768 raft_consensus.cc:3037] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:14.068905 28907 raft_consensus.cc:3037] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:14.071079 28768 raft_consensus.cc:2454] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c837dfccdf4dadad4ce7026d001677 in term 1.
I20260927 13:50:14.071581 28978 leader_election.cc:302] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c837dfccdf4dadad4ce7026d001677, fa030a9b1082446f8e6856dfaa240000; no voters:
I20260927 13:50:14.071580 28907 raft_consensus.cc:2454] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c837dfccdf4dadad4ce7026d001677 in term 1.
I20260927 13:50:14.071897 29127 raft_consensus.cc:2781] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:14.072517 29127 raft_consensus.cc:686] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 1 LEADER]: Becoming Leader. State: Replica: 07c837dfccdf4dadad4ce7026d001677, State: Running, Role: LEADER
I20260927 13:50:14.073058 29127 consensus_queue.cc:237] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.078539 28611 catalog_manager.cc:5495] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67). New cstate: current_term: 1 leader_uuid: "07c837dfccdf4dadad4ce7026d001677" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } health_report { overall_health: HEALTHY } } }
W20260927 13:50:14.085618 28819 tablet.cc:2369] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:50:14.101541 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.094756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.20.133.67:44015)
0927 13:50:14.094992 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:14.094997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:14.095106 (+ 109us) server_negotiation.cc:407] Connection header received
0927 13:50:14.095430 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:14.095435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:14.095551 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:14.095727 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:50:14.096246 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.097160 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.098721 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.099212 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.099365 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.099985 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:50:14.100852 (+ 867us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:50:14.100988 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:14.101146 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 13:50:14.101241 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 13:50:14.112509 28907 raft_consensus.cc:1261] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Refusing update from remote peer 07c837dfccdf4dadad4ce7026d001677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:14.112692 28768 raft_consensus.cc:1261] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Refusing update from remote peer 07c837dfccdf4dadad4ce7026d001677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:14.113721 29130 consensus_queue.cc:1035] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:14.114234 29127 consensus_queue.cc:1035] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:14.128376 29133 mvcc.cc:205] Tried to move back new op lower bound from 7333957689790693376 to 7333957689647378432. Current Snapshot: MvccSnapshot[applied={T|T < 7333957689790693376}]
I20260927 13:50:14.148917 28611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:14.149237 28611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:14.153903 28611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e: Sending DeleteTablet for 3 replicas of tablet fc7b3dcbe5f2441085f5caf1ab4fafd0
I20260927 13:50:14.155422 29026 tablet_service.cc:1514] Processing DeleteTablet for tablet fc7b3dcbe5f2441085f5caf1ab4fafd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:14 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:50972
I20260927 13:50:14.155826 28748 tablet_service.cc:1514] Processing DeleteTablet for tablet fc7b3dcbe5f2441085f5caf1ab4fafd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:14 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:58750
I20260927 13:50:14.156628 29143 tablet_replica.cc:330] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: stopping tablet replica
I20260927 13:50:14.156719 28887 tablet_service.cc:1514] Processing DeleteTablet for tablet fc7b3dcbe5f2441085f5caf1ab4fafd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:14 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:38558
I20260927 13:50:14.157450 29144 tablet_replica.cc:330] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: stopping tablet replica
I20260927 13:50:14.157488 29143 raft_consensus.cc:2229] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:50:14.157912 29144 raft_consensus.cc:2229] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:14.158403 29143 raft_consensus.cc:2258] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:14.158520 29144 raft_consensus.cc:2258] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:14.161286 29144 ts_tablet_manager.cc:1905] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:14.161239 29143 ts_tablet_manager.cc:1905] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:14.162593 28611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:50:14.163453 28611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:50:14.169502 29143 ts_tablet_manager.cc:1918] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:14.169906 29143 log.cc:1196] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/fc7b3dcbe5f2441085f5caf1ab4fafd0
I20260927 13:50:14.170621 29143 ts_tablet_manager.cc:1939] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 07c837dfccdf4dadad4ce7026d001677: Deleting consensus metadata
I20260927 13:50:14.172959 28595 catalog_manager.cc:4841] TS 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015): tablet fc7b3dcbe5f2441085f5caf1ab4fafd0 (table test-table [id=c4344d2cfba74f6d8acd90991a7b1534]) successfully deleted
I20260927 13:50:14.173938 29145 tablet_replica.cc:330] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: stopping tablet replica
I20260927 13:50:14.174474 29145 raft_consensus.cc:2229] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:14.174811 28886 tablet_service.cc:1467] Processing CreateTablet for tablet 1abdc670eb064b8aa8262ed8ce631c79 (DEFAULT_TABLE table=test-table [id=330ee5a7d4604b98ad26fae1ec96ad01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:14.175158 29145 raft_consensus.cc:2258] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:14.175565 28886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abdc670eb064b8aa8262ed8ce631c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:14.177990 29026 tablet_service.cc:1467] Processing CreateTablet for tablet 1abdc670eb064b8aa8262ed8ce631c79 (DEFAULT_TABLE table=test-table [id=330ee5a7d4604b98ad26fae1ec96ad01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:14.178589 29026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abdc670eb064b8aa8262ed8ce631c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:14.181300 28748 tablet_service.cc:1467] Processing CreateTablet for tablet 1abdc670eb064b8aa8262ed8ce631c79 (DEFAULT_TABLE table=test-table [id=330ee5a7d4604b98ad26fae1ec96ad01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:14.181902 28748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abdc670eb064b8aa8262ed8ce631c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:14.182799 29144 ts_tablet_manager.cc:1918] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:14.183070 29144 log.cc:1196] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/fc7b3dcbe5f2441085f5caf1ab4fafd0
I20260927 13:50:14.183807 29144 ts_tablet_manager.cc:1939] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P fa030a9b1082446f8e6856dfaa240000: Deleting consensus metadata
I20260927 13:50:14.186163 29121 tablet_bootstrap.cc:492] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677: Bootstrap starting.
I20260927 13:50:14.186293 28596 catalog_manager.cc:4841] TS fa030a9b1082446f8e6856dfaa240000 (127.20.133.65:36575): tablet fc7b3dcbe5f2441085f5caf1ab4fafd0 (table test-table [id=c4344d2cfba74f6d8acd90991a7b1534]) successfully deleted
I20260927 13:50:14.189025 29145 ts_tablet_manager.cc:1905] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:14.189954 29120 tablet_bootstrap.cc:492] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000: Bootstrap starting.
I20260927 13:50:14.190747 29121 tablet_bootstrap.cc:654] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:14.191713 29120 tablet_bootstrap.cc:654] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:14.192600 29122 tablet_bootstrap.cc:492] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82: Bootstrap starting.
I20260927 13:50:14.194304 29122 tablet_bootstrap.cc:654] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:14.196727 29145 ts_tablet_manager.cc:1918] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:14.196925 29145 log.cc:1196] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/fc7b3dcbe5f2441085f5caf1ab4fafd0
I20260927 13:50:14.197434 29145 ts_tablet_manager.cc:1939] T fc7b3dcbe5f2441085f5caf1ab4fafd0 P 337c0ed1b3e648a691bb711be0a84b82: Deleting consensus metadata
I20260927 13:50:14.199669 28596 catalog_manager.cc:4841] TS 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613): tablet fc7b3dcbe5f2441085f5caf1ab4fafd0 (table test-table [id=c4344d2cfba74f6d8acd90991a7b1534]) successfully deleted
I20260927 13:50:14.201946 29122 tablet_bootstrap.cc:492] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82: No bootstrap required, opened a new log
I20260927 13:50:14.202111 29122 ts_tablet_manager.cc:1397] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 13:50:14.202982 29120 tablet_bootstrap.cc:492] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000: No bootstrap required, opened a new log
I20260927 13:50:14.202986 29122 raft_consensus.cc:348] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.203143 29120 ts_tablet_manager.cc:1397] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20260927 13:50:14.203235 29122 raft_consensus.cc:374] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:14.203359 29122 raft_consensus.cc:729] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337c0ed1b3e648a691bb711be0a84b82, State: Initialized, Role: FOLLOWER
I20260927 13:50:14.203642 29121 tablet_bootstrap.cc:492] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677: No bootstrap required, opened a new log
I20260927 13:50:14.203666 29122 consensus_queue.cc:260] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.203866 29121 ts_tablet_manager.cc:1397] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.006s
I20260927 13:50:14.203981 29120 raft_consensus.cc:348] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.204213 29120 raft_consensus.cc:374] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:14.204319 29120 raft_consensus.cc:729] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa030a9b1082446f8e6856dfaa240000, State: Initialized, Role: FOLLOWER
I20260927 13:50:14.204602 29120 consensus_queue.cc:260] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.205320 29120 ts_tablet_manager.cc:1428] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 13:50:14.205886 29122 ts_tablet_manager.cc:1428] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260927 13:50:14.205691 29121 raft_consensus.cc:348] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.207334 29121 raft_consensus.cc:374] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:14.207459 29121 raft_consensus.cc:729] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c837dfccdf4dadad4ce7026d001677, State: Initialized, Role: FOLLOWER
I20260927 13:50:14.207831 29121 consensus_queue.cc:260] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.209723 29121 ts_tablet_manager.cc:1428] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260927 13:50:14.504062 29126 raft_consensus.cc:482] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:50:14.504396 29126 raft_consensus.cc:504] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.506480 29126 leader_election.cc:288] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613), 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015)
I20260927 13:50:14.515599 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.507504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:37359 (local address 127.20.133.66:40613)
0927 13:50:14.507808 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:14.507815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:14.508309 (+ 494us) server_negotiation.cc:407] Connection header received
0927 13:50:14.508675 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:14.508679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:14.508751 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:14.508841 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:14.509551 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.510581 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.513365 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.514576 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.514746 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.514821 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:14.515183 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 13:50:14.515315 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260927 13:50:14.515576 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.507238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:50057 (local address 127.20.133.67:44015)
0927 13:50:14.507603 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:14.507611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:14.508792 (+ 1181us) server_negotiation.cc:407] Connection header received
0927 13:50:14.509098 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:14.509104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:14.509192 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:14.509308 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:14.510031 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.511086 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.513067 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.514333 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.514513 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.514599 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:14.515045 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 13:50:14.515198 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260927 13:50:14.515589 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.507101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:44015 (local address 127.20.133.65:50057)
0927 13:50:14.508646 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:14.508668 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:14.508857 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:14.509372 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:14.509377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:14.509407 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:14.509844 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.509858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.511225 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.511230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.512901 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.512914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.514590 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.514597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.514837 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.514885 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 13:50:14.515000 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 13:50:14.515182 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1397,"thread_start_us":677,"threads_started":1}
I20260927 13:50:14.516516 29150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.507044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:40613 (local address 127.20.133.65:37359)
0927 13:50:14.508161 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:14.508187 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:14.508503 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:14.508906 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:14.508911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:14.508936 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:14.509381 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.509395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.510760 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.510766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.513180 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.513197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.514839 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.514846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.514993 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.515028 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:50:14.516025 (+ 997us) client_negotiation.cc:240] Negotiation successful
0927 13:50:14.516184 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":1200,"threads_started":1}
I20260927 13:50:14.517050 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abdc670eb064b8aa8262ed8ce631c79" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c837dfccdf4dadad4ce7026d001677" is_pre_election: true
I20260927 13:50:14.517350 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abdc670eb064b8aa8262ed8ce631c79" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337c0ed1b3e648a691bb711be0a84b82" is_pre_election: true
I20260927 13:50:14.517632 29046 raft_consensus.cc:2454] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 0.
I20260927 13:50:14.517663 28907 raft_consensus.cc:2454] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 0.
I20260927 13:50:14.518293 28699 leader_election.cc:302] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07c837dfccdf4dadad4ce7026d001677, 337c0ed1b3e648a691bb711be0a84b82, fa030a9b1082446f8e6856dfaa240000; no voters:
I20260927 13:50:14.518920 29126 raft_consensus.cc:2781] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:50:14.519075 29126 raft_consensus.cc:482] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:50:14.519183 29126 raft_consensus.cc:3037] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:14.522065 29126 raft_consensus.cc:504] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.522783 29126 leader_election.cc:288] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 election: Requested vote from peers 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613), 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015)
I20260927 13:50:14.523445 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abdc670eb064b8aa8262ed8ce631c79" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c837dfccdf4dadad4ce7026d001677"
I20260927 13:50:14.523500 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abdc670eb064b8aa8262ed8ce631c79" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337c0ed1b3e648a691bb711be0a84b82"
I20260927 13:50:14.523720 29046 raft_consensus.cc:3037] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:14.523768 28907 raft_consensus.cc:3037] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:14.526527 29046 raft_consensus.cc:2454] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 1.
I20260927 13:50:14.526916 28907 raft_consensus.cc:2454] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 1.
I20260927 13:50:14.527055 28698 leader_election.cc:302] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c837dfccdf4dadad4ce7026d001677, fa030a9b1082446f8e6856dfaa240000; no voters:
I20260927 13:50:14.527443 29126 raft_consensus.cc:2781] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:14.527976 29126 raft_consensus.cc:686] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 1 LEADER]: Becoming Leader. State: Replica: fa030a9b1082446f8e6856dfaa240000, State: Running, Role: LEADER
I20260927 13:50:14.528712 29126 consensus_queue.cc:237] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.533114 28611 catalog_manager.cc:5495] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa030a9b1082446f8e6856dfaa240000 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "fa030a9b1082446f8e6856dfaa240000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } health_report { overall_health: UNKNOWN } } }
Sep 27 13:50:14 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517011, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.126@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:50:14.603282 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.587209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:35867 (local address 127.0.0.1:52574)
0927 13:50:14.588112 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:14.588140 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:14.588357 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:14.588933 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:14.588937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:14.589644 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:14.589889 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.589899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.591612 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.591618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.592736 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.592751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.593630 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.593636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.593803 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.594559 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:14.594584 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:14.597043 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:14.600460 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:14.600471 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:14.600477 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:14.600943 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:14.601368 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:14.601375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:14.601377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:14.601587 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:14.602313 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:14.602321 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:14.602530 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 13:50:14.602703 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 13:50:14.602939 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":477,"threads_started":1}
I20260927 13:50:14.603609 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.587289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.20.133.126:35867)
0927 13:50:14.588499 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:14.588506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:14.588537 (+ 31us) server_negotiation.cc:407] Connection header received
0927 13:50:14.588610 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:14.588617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:14.588706 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:14.588852 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:14.590069 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.591418 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.592901 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.593415 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.593525 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.597308 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:14.597313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:14.597317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:14.597356 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:14.600252 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:14.601071 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:14.601075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:14.601078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:14.601224 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:14.601737 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:14.601742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:14.601745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:14.602170 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:14.602298 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:14.602885 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 13:50:14.603123 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":860,"threads_started":1}
I20260927 13:50:14.609983 28611 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:52574:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 13:50:14.611325 28611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:50:14.619364 28886 tablet_service.cc:1467] Processing CreateTablet for tablet e3e184a16ff649b58b96fcbd14f4b73d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd310fddde244a22a835651b496047f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:50:14.619463 28748 tablet_service.cc:1467] Processing CreateTablet for tablet e3e184a16ff649b58b96fcbd14f4b73d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd310fddde244a22a835651b496047f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:50:14.619611 29026 tablet_service.cc:1467] Processing CreateTablet for tablet e3e184a16ff649b58b96fcbd14f4b73d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd310fddde244a22a835651b496047f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:50:14.619978 28886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e184a16ff649b58b96fcbd14f4b73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:14.620070 28748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e184a16ff649b58b96fcbd14f4b73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:14.620173 29026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e184a16ff649b58b96fcbd14f4b73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:14.627017 29120 tablet_bootstrap.cc:492] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: Bootstrap starting.
I20260927 13:50:14.627310 29121 tablet_bootstrap.cc:492] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677: Bootstrap starting.
I20260927 13:50:14.627837 29122 tablet_bootstrap.cc:492] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82: Bootstrap starting.
I20260927 13:50:14.629002 29120 tablet_bootstrap.cc:654] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:14.629202 29121 tablet_bootstrap.cc:654] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:14.629707 29122 tablet_bootstrap.cc:654] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:14.631405 29121 tablet_bootstrap.cc:492] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677: No bootstrap required, opened a new log
I20260927 13:50:14.631567 29121 ts_tablet_manager.cc:1397] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 13:50:14.631800 29120 tablet_bootstrap.cc:492] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: No bootstrap required, opened a new log
I20260927 13:50:14.631966 29120 ts_tablet_manager.cc:1397] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260927 13:50:14.632390 29121 raft_consensus.cc:348] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.632618 29121 raft_consensus.cc:374] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:14.632710 29121 raft_consensus.cc:729] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c837dfccdf4dadad4ce7026d001677, State: Initialized, Role: FOLLOWER
I20260927 13:50:14.632812 29120 raft_consensus.cc:348] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.633093 29120 raft_consensus.cc:374] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:14.633178 29120 raft_consensus.cc:729] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa030a9b1082446f8e6856dfaa240000, State: Initialized, Role: FOLLOWER
I20260927 13:50:14.633203 29122 tablet_bootstrap.cc:492] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82: No bootstrap required, opened a new log
I20260927 13:50:14.633067 29121 consensus_queue.cc:260] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.633371 29122 ts_tablet_manager.cc:1397] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 13:50:14.633661 29121 ts_tablet_manager.cc:1428] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 13:50:14.633512 29120 consensus_queue.cc:260] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.633581 29127 tablet_replica.cc:439] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } } }
I20260927 13:50:14.633857 29127 tablet_replica.cc:442] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:50:14.633956 29126 tablet_replica.cc:439] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } } }
I20260927 13:50:14.634256 29126 tablet_replica.cc:442] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:50:14.634214 29122 raft_consensus.cc:348] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.634466 29122 raft_consensus.cc:374] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:14.634826 29122 raft_consensus.cc:729] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337c0ed1b3e648a691bb711be0a84b82, State: Initialized, Role: FOLLOWER
I20260927 13:50:14.635480 29120 ts_tablet_manager.cc:1428] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 13:50:14.635955 29122 consensus_queue.cc:260] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.636838 29128 tablet_replica.cc:439] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } } }
I20260927 13:50:14.637300 29128 tablet_replica.cc:442] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:50:14.637413 29122 ts_tablet_manager.cc:1428] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260927 13:50:14.764142 29126 raft_consensus.cc:482] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:50:14.764647 29126 raft_consensus.cc:504] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.765493 29126 leader_election.cc:288] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613), 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015)
I20260927 13:50:14.766276 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e184a16ff649b58b96fcbd14f4b73d" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337c0ed1b3e648a691bb711be0a84b82" is_pre_election: true
I20260927 13:50:14.766284 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e184a16ff649b58b96fcbd14f4b73d" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c837dfccdf4dadad4ce7026d001677" is_pre_election: true
I20260927 13:50:14.766561 29046 raft_consensus.cc:2454] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 0.
I20260927 13:50:14.766594 28907 raft_consensus.cc:2454] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 0.
I20260927 13:50:14.767128 28699 leader_election.cc:302] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 337c0ed1b3e648a691bb711be0a84b82, fa030a9b1082446f8e6856dfaa240000; no voters:
I20260927 13:50:14.767575 29126 raft_consensus.cc:2781] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:50:14.767695 29126 raft_consensus.cc:482] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:50:14.767786 29126 raft_consensus.cc:3037] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:14.770284 29126 raft_consensus.cc:504] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.770991 29126 leader_election.cc:288] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 election: Requested vote from peers 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613), 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015)
I20260927 13:50:14.771791 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e184a16ff649b58b96fcbd14f4b73d" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337c0ed1b3e648a691bb711be0a84b82"
I20260927 13:50:14.771835 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e184a16ff649b58b96fcbd14f4b73d" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c837dfccdf4dadad4ce7026d001677"
I20260927 13:50:14.772119 29046 raft_consensus.cc:3037] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:14.772123 28907 raft_consensus.cc:3037] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:14.774197 28907 raft_consensus.cc:2454] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 1.
I20260927 13:50:14.774529 29046 raft_consensus.cc:2454] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 1.
I20260927 13:50:14.774771 28699 leader_election.cc:302] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 337c0ed1b3e648a691bb711be0a84b82, fa030a9b1082446f8e6856dfaa240000; no voters:
I20260927 13:50:14.775153 29126 raft_consensus.cc:2781] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:14.775367 29126 raft_consensus.cc:686] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [term 1 LEADER]: Becoming Leader. State: Replica: fa030a9b1082446f8e6856dfaa240000, State: Running, Role: LEADER
I20260927 13:50:14.775684 29126 consensus_queue.cc:237] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:14.777773 29151 tablet_replica.cc:439] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: TxnStatusTablet state changed. Reason: New leader fa030a9b1082446f8e6856dfaa240000. Latest consensus state: current_term: 1 leader_uuid: "fa030a9b1082446f8e6856dfaa240000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } } }
I20260927 13:50:14.778044 29151 tablet_replica.cc:442] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:50:14.778707 29164 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 13:50:14.779014 29164 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 13:50:14.779415 28609 catalog_manager.cc:5495] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa030a9b1082446f8e6856dfaa240000 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "fa030a9b1082446f8e6856dfaa240000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } health_report { overall_health: UNKNOWN } } }
I20260927 13:50:14.834802 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.828057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36575 (local address 127.0.0.1:58766)
0927 13:50:14.828354 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:14.828378 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:14.828555 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:14.829308 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:14.829313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:14.829327 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:50:14.829542 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.829552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.830818 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.830824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.831923 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.831936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.832807 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.832813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.832987 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.833057 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:50:14.834153 (+ 1096us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:50:14.834179 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 13:50:14.834274 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 13:50:14.834460 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 13:50:14.834879 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.828112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.20.133.65:36575)
0927 13:50:14.828879 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:14.828884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:14.828900 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:50:14.829005 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:14.829009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:14.829102 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:14.829253 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:50:14.829712 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.830681 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.832072 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.832631 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.832770 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.833277 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:50:14.834044 (+ 767us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:50:14.834189 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:14.834323 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 13:50:14.834428 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":361,"threads_started":1}
I20260927 13:50:14.840963 28907 raft_consensus.cc:1261] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Refusing update from remote peer fa030a9b1082446f8e6856dfaa240000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:14.841287 29046 raft_consensus.cc:1261] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Refusing update from remote peer fa030a9b1082446f8e6856dfaa240000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:14.841814 29151 consensus_queue.cc:1035] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:14.842221 29126 consensus_queue.cc:1035] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:14.850328 29128 tablet_replica.cc:439] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82: TxnStatusTablet state changed. Reason: New leader fa030a9b1082446f8e6856dfaa240000. Latest consensus state: current_term: 1 leader_uuid: "fa030a9b1082446f8e6856dfaa240000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } } }
I20260927 13:50:14.850692 29128 tablet_replica.cc:442] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:50:14.853622 29168 mvcc.cc:205] Tried to move back new op lower bound from 7333957692778815488 to 7333957692524691456. Current Snapshot: MvccSnapshot[applied={T|T < 7333957692778815488}]
I20260927 13:50:14.854822 29151 tablet_replica.cc:439] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa030a9b1082446f8e6856dfaa240000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } } }
I20260927 13:50:14.855165 29151 tablet_replica.cc:442] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:50:14.869836 29127 tablet_replica.cc:439] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677: TxnStatusTablet state changed. Reason: New leader fa030a9b1082446f8e6856dfaa240000. Latest consensus state: current_term: 1 leader_uuid: "fa030a9b1082446f8e6856dfaa240000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } } }
I20260927 13:50:14.870164 29127 tablet_replica.cc:442] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:50:14.872816 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.863983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.20.133.65:36575)
0927 13:50:14.864197 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:14.864204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:14.864806 (+ 602us) server_negotiation.cc:407] Connection header received
0927 13:50:14.865149 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:14.865157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:14.865277 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:14.865369 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:50:14.866223 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.867654 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.869901 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.870677 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.870789 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.871689 (+ 900us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:50:14.871871 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:50:14.871935 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:14.872184 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 13:50:14.872301 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 13:50:14.874013 29128 tablet_replica.cc:439] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa030a9b1082446f8e6856dfaa240000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } } }
I20260927 13:50:14.874356 29128 tablet_replica.cc:442] T e3e184a16ff649b58b96fcbd14f4b73d P 337c0ed1b3e648a691bb711be0a84b82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:50:14.877450 29177 tablet_replica.cc:439] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa030a9b1082446f8e6856dfaa240000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } } }
I20260927 13:50:14.879913 29177 tablet_replica.cc:442] T e3e184a16ff649b58b96fcbd14f4b73d P fa030a9b1082446f8e6856dfaa240000: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:50:14.880904 29127 tablet_replica.cc:439] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa030a9b1082446f8e6856dfaa240000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } } }
I20260927 13:50:14.881253 29127 tablet_replica.cc:442] T e3e184a16ff649b58b96fcbd14f4b73d P 07c837dfccdf4dadad4ce7026d001677: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:50:14.901337 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.893196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:41415 (local address 127.20.133.65:36575)
0927 13:50:14.893472 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:14.893479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:14.894258 (+ 779us) server_negotiation.cc:407] Connection header received
0927 13:50:14.894478 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:14.894484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:14.894623 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:14.894774 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:50:14.895383 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.896601 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.898578 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.899338 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.899556 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.900282 (+ 726us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:50:14.900509 (+ 227us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:50:14.900610 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:14.900809 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 13:50:14.900928 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 13:50:14.901459 29181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:14.893214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:36575 (local address 127.20.133.65:41415)
0927 13:50:14.894130 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:14.894148 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:14.894312 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:14.894793 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:14.894797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:14.894808 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:50:14.895205 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.895214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.896775 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.896779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.898393 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:14.898408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:14.899762 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:14.899770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:14.899954 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:14.900044 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:50:14.900653 (+ 609us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:50:14.900674 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 13:50:14.900765 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 13:50:14.900957 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":496,"threads_started":1}
I20260927 13:50:14.914690 28907 raft_consensus.cc:1261] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Refusing update from remote peer fa030a9b1082446f8e6856dfaa240000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:14.914676 29046 raft_consensus.cc:1261] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Refusing update from remote peer fa030a9b1082446f8e6856dfaa240000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:14.915325 29126 consensus_queue.cc:1035] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:14.915692 29151 consensus_queue.cc:1035] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:14.991106 28609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:14.991483 28609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:14.995610 28609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e: Sending DeleteTablet for 3 replicas of tablet 1abdc670eb064b8aa8262ed8ce631c79
I20260927 13:50:14.996728 28886 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abdc670eb064b8aa8262ed8ce631c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:14 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:38558
I20260927 13:50:14.996723 28747 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abdc670eb064b8aa8262ed8ce631c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:14 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:58750
I20260927 13:50:14.997236 29026 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abdc670eb064b8aa8262ed8ce631c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:14 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:50972
I20260927 13:50:14.999823 29187 tablet_replica.cc:330] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000: stopping tablet replica
I20260927 13:50:15.000098 29187 raft_consensus.cc:2229] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:50:15.000878 29188 tablet_replica.cc:330] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677: stopping tablet replica
I20260927 13:50:15.000847 29187 raft_consensus.cc:2258] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:15.001307 29188 raft_consensus.cc:2229] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:15.001636 29188 raft_consensus.cc:2258] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:15.002494 29187 ts_tablet_manager.cc:1905] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:15.003105 29188 ts_tablet_manager.cc:1905] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:15.012140 29186 tablet_replica.cc:330] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82: stopping tablet replica
I20260927 13:50:15.012418 29186 raft_consensus.cc:2229] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:15.012445 29187 ts_tablet_manager.cc:1918] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:50:15.012594 29187 log.cc:1196] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/1abdc670eb064b8aa8262ed8ce631c79
I20260927 13:50:15.012705 29186 raft_consensus.cc:2258] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:15.013120 29187 ts_tablet_manager.cc:1939] T 1abdc670eb064b8aa8262ed8ce631c79 P fa030a9b1082446f8e6856dfaa240000: Deleting consensus metadata
I20260927 13:50:15.013617 29188 ts_tablet_manager.cc:1918] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:50:15.013855 29188 log.cc:1196] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/1abdc670eb064b8aa8262ed8ce631c79
I20260927 13:50:15.014460 29188 ts_tablet_manager.cc:1939] T 1abdc670eb064b8aa8262ed8ce631c79 P 07c837dfccdf4dadad4ce7026d001677: Deleting consensus metadata
I20260927 13:50:15.014564 28596 catalog_manager.cc:4841] TS fa030a9b1082446f8e6856dfaa240000 (127.20.133.65:36575): tablet 1abdc670eb064b8aa8262ed8ce631c79 (table test-table [id=330ee5a7d4604b98ad26fae1ec96ad01]) successfully deleted
I20260927 13:50:15.015687 29186 ts_tablet_manager.cc:1905] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:15.016022 28595 catalog_manager.cc:4841] TS 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015): tablet 1abdc670eb064b8aa8262ed8ce631c79 (table test-table [id=330ee5a7d4604b98ad26fae1ec96ad01]) successfully deleted
I20260927 13:50:15.023116 29186 ts_tablet_manager.cc:1918] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:50:15.023306 29186 log.cc:1196] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/1abdc670eb064b8aa8262ed8ce631c79
I20260927 13:50:15.023717 29186 ts_tablet_manager.cc:1939] T 1abdc670eb064b8aa8262ed8ce631c79 P 337c0ed1b3e648a691bb711be0a84b82: Deleting consensus metadata
I20260927 13:50:15.024959 28596 catalog_manager.cc:4841] TS 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613): tablet 1abdc670eb064b8aa8262ed8ce631c79 (table test-table [id=330ee5a7d4604b98ad26fae1ec96ad01]) successfully deleted
Sep 27 13:50:15 dist-test-slave-q25r krb5kdc[28566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:50:15 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:15 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:50:15.072826 29159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:15.059142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.20.133.126:35867)
0927 13:50:15.059422 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:15.059430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:15.059755 (+ 325us) server_negotiation.cc:407] Connection header received
0927 13:50:15.059953 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:15.059960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:15.060090 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:15.060258 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:15.062394 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.063692 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.065334 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.066055 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.066216 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:15.072562 (+ 6346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52588: BlockingRecv error: recv got EOF from 127.0.0.1:52588 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 13:50:15 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM
I20260927 13:50:15.095821 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:15.076533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.20.133.126:35867)
0927 13:50:15.076751 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:15.076758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:15.077087 (+ 329us) server_negotiation.cc:407] Connection header received
0927 13:50:15.077318 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:15.077325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:15.077428 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:15.077551 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:15.079539 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.080760 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.082313 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.082992 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.083136 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:15.087675 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:15.087683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:15.087689 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:15.087753 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:15.091774 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:15.092879 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:15.092885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:15.092889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:15.093101 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:15.093816 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:15.093822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:15.093826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:15.094319 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:15.094648 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:15.095067 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 13:50:15.095388 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:50:15.101249 28609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:50:15.101963 28609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:50:15.110710 28747 tablet_service.cc:1467] Processing CreateTablet for tablet 65da67bd1b934e409e89aa9f60c92612 (DEFAULT_TABLE table=test-table [id=341636ee80a14b4fa2999962b29f8fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:15.111388 28747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65da67bd1b934e409e89aa9f60c92612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:15.113641 29026 tablet_service.cc:1467] Processing CreateTablet for tablet 65da67bd1b934e409e89aa9f60c92612 (DEFAULT_TABLE table=test-table [id=341636ee80a14b4fa2999962b29f8fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:15.114138 29026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65da67bd1b934e409e89aa9f60c92612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:15.115115 28886 tablet_service.cc:1467] Processing CreateTablet for tablet 65da67bd1b934e409e89aa9f60c92612 (DEFAULT_TABLE table=test-table [id=341636ee80a14b4fa2999962b29f8fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:15.115684 28886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65da67bd1b934e409e89aa9f60c92612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:15.120803 29121 tablet_bootstrap.cc:492] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677: Bootstrap starting.
I20260927 13:50:15.122419 29121 tablet_bootstrap.cc:654] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:15.128732 29122 tablet_bootstrap.cc:492] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82: Bootstrap starting.
I20260927 13:50:15.130908 29122 tablet_bootstrap.cc:654] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:15.136409 29120 tablet_bootstrap.cc:492] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000: Bootstrap starting.
I20260927 13:50:15.138500 29120 tablet_bootstrap.cc:654] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:15.141338 29122 tablet_bootstrap.cc:492] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82: No bootstrap required, opened a new log
I20260927 13:50:15.141659 29122 ts_tablet_manager.cc:1397] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20260927 13:50:15.142635 29122 raft_consensus.cc:348] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.143049 29122 raft_consensus.cc:374] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:15.143200 29122 raft_consensus.cc:729] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337c0ed1b3e648a691bb711be0a84b82, State: Initialized, Role: FOLLOWER
I20260927 13:50:15.143572 29122 consensus_queue.cc:260] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.148595 29120 tablet_bootstrap.cc:492] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000: No bootstrap required, opened a new log
I20260927 13:50:15.148829 29120 ts_tablet_manager.cc:1397] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20260927 13:50:15.149451 29122 ts_tablet_manager.cc:1428] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20260927 13:50:15.149889 29120 raft_consensus.cc:348] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.154299 29120 raft_consensus.cc:374] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:15.154446 29120 raft_consensus.cc:729] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa030a9b1082446f8e6856dfaa240000, State: Initialized, Role: FOLLOWER
I20260927 13:50:15.154812 29120 consensus_queue.cc:260] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.157202 29120 ts_tablet_manager.cc:1428] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260927 13:50:15.159941 29121 tablet_bootstrap.cc:492] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677: No bootstrap required, opened a new log
I20260927 13:50:15.160152 29121 ts_tablet_manager.cc:1397] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.020s
I20260927 13:50:15.161015 29121 raft_consensus.cc:348] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.161273 29121 raft_consensus.cc:374] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:15.161396 29121 raft_consensus.cc:729] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c837dfccdf4dadad4ce7026d001677, State: Initialized, Role: FOLLOWER
I20260927 13:50:15.161664 29121 consensus_queue.cc:260] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.162263 29121 ts_tablet_manager.cc:1428] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 13:50:15.166515 29184 raft_consensus.cc:482] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:50:15.166757 29184 raft_consensus.cc:504] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.167601 29184 leader_election.cc:288] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613), 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015)
I20260927 13:50:15.168143 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65da67bd1b934e409e89aa9f60c92612" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337c0ed1b3e648a691bb711be0a84b82" is_pre_election: true
I20260927 13:50:15.168260 29045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65da67bd1b934e409e89aa9f60c92612" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c837dfccdf4dadad4ce7026d001677" is_pre_election: true
I20260927 13:50:15.168471 28907 raft_consensus.cc:2454] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 0.
I20260927 13:50:15.168536 29045 raft_consensus.cc:2454] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 0.
I20260927 13:50:15.169008 28698 leader_election.cc:302] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c837dfccdf4dadad4ce7026d001677, fa030a9b1082446f8e6856dfaa240000; no voters:
I20260927 13:50:15.169405 29184 raft_consensus.cc:2781] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:50:15.169600 29184 raft_consensus.cc:482] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:50:15.169706 29184 raft_consensus.cc:3037] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:15.173579 29184 raft_consensus.cc:504] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.174588 29184 leader_election.cc:288] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 election: Requested vote from peers 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613), 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015)
I20260927 13:50:15.175367 29045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65da67bd1b934e409e89aa9f60c92612" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c837dfccdf4dadad4ce7026d001677"
I20260927 13:50:15.175676 29045 raft_consensus.cc:3037] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:15.176692 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65da67bd1b934e409e89aa9f60c92612" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337c0ed1b3e648a691bb711be0a84b82"
I20260927 13:50:15.176963 28907 raft_consensus.cc:3037] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:15.179193 29045 raft_consensus.cc:2454] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 1.
I20260927 13:50:15.179615 28907 raft_consensus.cc:2454] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 1.
I20260927 13:50:15.180065 28698 leader_election.cc:302] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c837dfccdf4dadad4ce7026d001677, fa030a9b1082446f8e6856dfaa240000; no voters:
I20260927 13:50:15.180593 29184 raft_consensus.cc:2781] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:15.180812 29184 raft_consensus.cc:686] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 1 LEADER]: Becoming Leader. State: Replica: fa030a9b1082446f8e6856dfaa240000, State: Running, Role: LEADER
I20260927 13:50:15.181174 29184 consensus_queue.cc:237] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.184832 28609 catalog_manager.cc:5495] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa030a9b1082446f8e6856dfaa240000 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "fa030a9b1082446f8e6856dfaa240000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } health_report { overall_health: UNKNOWN } } }
I20260927 13:50:15.205411 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:15.198911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.20.133.65:36575)
0927 13:50:15.199114 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:15.199120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:15.199320 (+ 200us) server_negotiation.cc:407] Connection header received
0927 13:50:15.199581 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:15.199586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:15.199671 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:15.199763 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:50:15.200432 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.201242 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.203322 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.203818 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.203921 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:15.204586 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:50:15.204752 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:50:15.204838 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:15.205032 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 13:50:15.205110 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:50:15.209013 29045 raft_consensus.cc:1261] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Refusing update from remote peer fa030a9b1082446f8e6856dfaa240000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:15.209025 28907 raft_consensus.cc:1261] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Refusing update from remote peer fa030a9b1082446f8e6856dfaa240000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:15.210005 29184 consensus_queue.cc:1035] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:15.210343 29177 consensus_queue.cc:1035] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:15.228603 28609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:15.228901 28609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:15.233314 28609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e: Sending DeleteTablet for 3 replicas of tablet 65da67bd1b934e409e89aa9f60c92612
I20260927 13:50:15.234714 29026 tablet_service.cc:1514] Processing DeleteTablet for tablet 65da67bd1b934e409e89aa9f60c92612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:15 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:50972
I20260927 13:50:15.234781 28886 tablet_service.cc:1514] Processing DeleteTablet for tablet 65da67bd1b934e409e89aa9f60c92612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:15 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:38558
I20260927 13:50:15.235090 29188 tablet_replica.cc:330] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677: stopping tablet replica
I20260927 13:50:15.235097 29186 tablet_replica.cc:330] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82: stopping tablet replica
I20260927 13:50:15.235316 29186 raft_consensus.cc:2229] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:15.235325 29188 raft_consensus.cc:2229] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:15.235590 29186 raft_consensus.cc:2258] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:15.235590 29188 raft_consensus.cc:2258] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:15.235978 28747 tablet_service.cc:1514] Processing DeleteTablet for tablet 65da67bd1b934e409e89aa9f60c92612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:15 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:58750
I20260927 13:50:15.236372 29187 tablet_replica.cc:330] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000: stopping tablet replica
I20260927 13:50:15.236637 29187 raft_consensus.cc:2229] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:50:15.237095 29187 raft_consensus.cc:2258] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:15.237365 29188 ts_tablet_manager.cc:1905] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:15.238652 29186 ts_tablet_manager.cc:1905] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:15.244854 29188 ts_tablet_manager.cc:1918] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:15.245502 29186 ts_tablet_manager.cc:1918] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:15.245234 28609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 13:50:15.245669 29186 log.cc:1196] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/65da67bd1b934e409e89aa9f60c92612
I20260927 13:50:15.245941 29188 log.cc:1196] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/65da67bd1b934e409e89aa9f60c92612
I20260927 13:50:15.246086 29186 ts_tablet_manager.cc:1939] T 65da67bd1b934e409e89aa9f60c92612 P 337c0ed1b3e648a691bb711be0a84b82: Deleting consensus metadata
W20260927 13:50:15.246366 28609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:50:15.246790 29188 ts_tablet_manager.cc:1939] T 65da67bd1b934e409e89aa9f60c92612 P 07c837dfccdf4dadad4ce7026d001677: Deleting consensus metadata
I20260927 13:50:15.247066 29187 ts_tablet_manager.cc:1905] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:15.248986 28595 catalog_manager.cc:4841] TS 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015): tablet 65da67bd1b934e409e89aa9f60c92612 (table test-table [id=341636ee80a14b4fa2999962b29f8fc9]) successfully deleted
I20260927 13:50:15.252194 28596 catalog_manager.cc:4841] TS 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613): tablet 65da67bd1b934e409e89aa9f60c92612 (table test-table [id=341636ee80a14b4fa2999962b29f8fc9]) successfully deleted
I20260927 13:50:15.254427 29187 ts_tablet_manager.cc:1918] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:15.254627 29187 log.cc:1196] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/65da67bd1b934e409e89aa9f60c92612
I20260927 13:50:15.255074 29187 ts_tablet_manager.cc:1939] T 65da67bd1b934e409e89aa9f60c92612 P fa030a9b1082446f8e6856dfaa240000: Deleting consensus metadata
I20260927 13:50:15.256560 28596 catalog_manager.cc:4841] TS fa030a9b1082446f8e6856dfaa240000 (127.20.133.65:36575): tablet 65da67bd1b934e409e89aa9f60c92612 (table test-table [id=341636ee80a14b4fa2999962b29f8fc9]) successfully deleted
I20260927 13:50:15.257508 28747 tablet_service.cc:1467] Processing CreateTablet for tablet 11115ffcbd1d472bbe9bb184727ad577 (DEFAULT_TABLE table=test-table [id=6addd083ba4f4a3ba4ff90f5b4c34858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:15.257480 29026 tablet_service.cc:1467] Processing CreateTablet for tablet 11115ffcbd1d472bbe9bb184727ad577 (DEFAULT_TABLE table=test-table [id=6addd083ba4f4a3ba4ff90f5b4c34858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:15.257980 28747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11115ffcbd1d472bbe9bb184727ad577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:15.257990 29026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11115ffcbd1d472bbe9bb184727ad577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:15.260900 28886 tablet_service.cc:1467] Processing CreateTablet for tablet 11115ffcbd1d472bbe9bb184727ad577 (DEFAULT_TABLE table=test-table [id=6addd083ba4f4a3ba4ff90f5b4c34858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:15.261448 28886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11115ffcbd1d472bbe9bb184727ad577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:15.265077 29121 tablet_bootstrap.cc:492] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677: Bootstrap starting.
I20260927 13:50:15.265084 29120 tablet_bootstrap.cc:492] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000: Bootstrap starting.
I20260927 13:50:15.267042 29121 tablet_bootstrap.cc:654] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:15.267896 29120 tablet_bootstrap.cc:654] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:15.269696 29122 tablet_bootstrap.cc:492] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82: Bootstrap starting.
I20260927 13:50:15.271448 29122 tablet_bootstrap.cc:654] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:15.275127 29122 tablet_bootstrap.cc:492] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82: No bootstrap required, opened a new log
I20260927 13:50:15.275369 29122 ts_tablet_manager.cc:1397] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 13:50:15.276393 29122 raft_consensus.cc:348] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.276697 29122 raft_consensus.cc:374] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:15.276764 29122 raft_consensus.cc:729] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337c0ed1b3e648a691bb711be0a84b82, State: Initialized, Role: FOLLOWER
I20260927 13:50:15.277107 29120 tablet_bootstrap.cc:492] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000: No bootstrap required, opened a new log
I20260927 13:50:15.277060 29122 consensus_queue.cc:260] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.277321 29120 ts_tablet_manager.cc:1397] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260927 13:50:15.278219 29120 raft_consensus.cc:348] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.278501 29120 raft_consensus.cc:374] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:15.278574 29120 raft_consensus.cc:729] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa030a9b1082446f8e6856dfaa240000, State: Initialized, Role: FOLLOWER
I20260927 13:50:15.278810 29120 consensus_queue.cc:260] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.280539 29120 ts_tablet_manager.cc:1428] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 13:50:15.281705 29122 ts_tablet_manager.cc:1428] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260927 13:50:15.282488 29121 tablet_bootstrap.cc:492] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677: No bootstrap required, opened a new log
I20260927 13:50:15.282711 29121 ts_tablet_manager.cc:1397] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260927 13:50:15.283656 29121 raft_consensus.cc:348] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.283957 29121 raft_consensus.cc:374] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:15.284040 29121 raft_consensus.cc:729] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c837dfccdf4dadad4ce7026d001677, State: Initialized, Role: FOLLOWER
I20260927 13:50:15.285090 29121 consensus_queue.cc:260] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.285841 29121 ts_tablet_manager.cc:1428] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 13:50:15.357954 29184 raft_consensus.cc:482] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:50:15.358206 29184 raft_consensus.cc:504] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.359009 29184 leader_election.cc:288] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613), 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015)
I20260927 13:50:15.359787 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11115ffcbd1d472bbe9bb184727ad577" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337c0ed1b3e648a691bb711be0a84b82" is_pre_election: true
I20260927 13:50:15.359798 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11115ffcbd1d472bbe9bb184727ad577" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c837dfccdf4dadad4ce7026d001677" is_pre_election: true
I20260927 13:50:15.360067 28907 raft_consensus.cc:2454] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 0.
I20260927 13:50:15.360141 29046 raft_consensus.cc:2454] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 0.
I20260927 13:50:15.360600 28699 leader_election.cc:302] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 337c0ed1b3e648a691bb711be0a84b82, fa030a9b1082446f8e6856dfaa240000; no voters:
I20260927 13:50:15.360960 29184 raft_consensus.cc:2781] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:50:15.361106 29184 raft_consensus.cc:482] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:50:15.361204 29184 raft_consensus.cc:3037] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:15.364817 29184 raft_consensus.cc:504] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.365629 29184 leader_election.cc:288] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 election: Requested vote from peers 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613), 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015)
I20260927 13:50:15.366456 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11115ffcbd1d472bbe9bb184727ad577" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c837dfccdf4dadad4ce7026d001677"
I20260927 13:50:15.366417 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11115ffcbd1d472bbe9bb184727ad577" candidate_uuid: "fa030a9b1082446f8e6856dfaa240000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337c0ed1b3e648a691bb711be0a84b82"
I20260927 13:50:15.366736 29046 raft_consensus.cc:3037] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:15.366737 28907 raft_consensus.cc:3037] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:15.370205 28907 raft_consensus.cc:2454] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 1.
I20260927 13:50:15.370205 29046 raft_consensus.cc:2454] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa030a9b1082446f8e6856dfaa240000 in term 1.
I20260927 13:50:15.370914 28698 leader_election.cc:302] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07c837dfccdf4dadad4ce7026d001677, 337c0ed1b3e648a691bb711be0a84b82, fa030a9b1082446f8e6856dfaa240000; no voters:
I20260927 13:50:15.371395 29184 raft_consensus.cc:2781] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:15.371610 29184 raft_consensus.cc:686] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 1 LEADER]: Becoming Leader. State: Replica: fa030a9b1082446f8e6856dfaa240000, State: Running, Role: LEADER
I20260927 13:50:15.371950 29184 consensus_queue.cc:237] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } }
I20260927 13:50:15.375429 28610 catalog_manager.cc:5495] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa030a9b1082446f8e6856dfaa240000 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "fa030a9b1082446f8e6856dfaa240000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa030a9b1082446f8e6856dfaa240000" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 36575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 } health_report { overall_health: UNKNOWN } } }
I20260927 13:50:15.428625 28907 raft_consensus.cc:1261] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Refusing update from remote peer fa030a9b1082446f8e6856dfaa240000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:15.429243 29046 raft_consensus.cc:1261] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Refusing update from remote peer fa030a9b1082446f8e6856dfaa240000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:15.429373 29184 consensus_queue.cc:1035] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "337c0ed1b3e648a691bb711be0a84b82" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:15.429898 29126 consensus_queue.cc:1035] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c837dfccdf4dadad4ce7026d001677" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:15.490159 28609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:15.490455 28609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:15.494567 28609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3457716c0554f75b221da3536bee34e: Sending DeleteTablet for 3 replicas of tablet 11115ffcbd1d472bbe9bb184727ad577
I20260927 13:50:15.495697 29026 tablet_service.cc:1514] Processing DeleteTablet for tablet 11115ffcbd1d472bbe9bb184727ad577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:15 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:50972
I20260927 13:50:15.495791 28748 tablet_service.cc:1514] Processing DeleteTablet for tablet 11115ffcbd1d472bbe9bb184727ad577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:15 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:58750
I20260927 13:50:15.496254 29187 tablet_replica.cc:330] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000: stopping tablet replica
I20260927 13:50:15.496284 29188 tablet_replica.cc:330] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677: stopping tablet replica
I20260927 13:50:15.496512 29187 raft_consensus.cc:2229] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:50:15.496532 29188 raft_consensus.cc:2229] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:15.496789 29188 raft_consensus.cc:2258] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:15.496920 29187 raft_consensus.cc:2258] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:15.497361 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 28682
I20260927 13:50:15.498250 29187 ts_tablet_manager.cc:1905] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:15.498459 29188 ts_tablet_manager.cc:1905] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:15.503029 28886 tablet_service.cc:1514] Processing DeleteTablet for tablet 11115ffcbd1d472bbe9bb184727ad577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:15 UTC) from {username='oryx', principal='oryx/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:38558
I20260927 13:50:15.503404 29186 tablet_replica.cc:330] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82: stopping tablet replica
I20260927 13:50:15.503579 29186 raft_consensus.cc:2229] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:15.503796 29186 raft_consensus.cc:2258] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:15.507297 29188 ts_tablet_manager.cc:1918] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:50:15.507905 29188 log.cc:1196] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/11115ffcbd1d472bbe9bb184727ad577
I20260927 13:50:15.507989 29187 ts_tablet_manager.cc:1918] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:50:15.508180 29187 log.cc:1196] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/11115ffcbd1d472bbe9bb184727ad577
I20260927 13:50:15.508677 29188 ts_tablet_manager.cc:1939] T 11115ffcbd1d472bbe9bb184727ad577 P 07c837dfccdf4dadad4ce7026d001677: Deleting consensus metadata
I20260927 13:50:15.508710 29187 ts_tablet_manager.cc:1939] T 11115ffcbd1d472bbe9bb184727ad577 P fa030a9b1082446f8e6856dfaa240000: Deleting consensus metadata
I20260927 13:50:15.510154 28596 catalog_manager.cc:4841] TS fa030a9b1082446f8e6856dfaa240000 (127.20.133.65:36575): tablet 11115ffcbd1d472bbe9bb184727ad577 (table test-table [id=6addd083ba4f4a3ba4ff90f5b4c34858]) successfully deleted
I20260927 13:50:15.510682 29186 ts_tablet_manager.cc:1905] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:15.510811 28595 catalog_manager.cc:4841] TS 07c837dfccdf4dadad4ce7026d001677 (127.20.133.67:44015): tablet 11115ffcbd1d472bbe9bb184727ad577 (table test-table [id=6addd083ba4f4a3ba4ff90f5b4c34858]) successfully deleted
I20260927 13:50:15.517753 29186 ts_tablet_manager.cc:1918] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:50:15.517930 29186 log.cc:1196] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/11115ffcbd1d472bbe9bb184727ad577
I20260927 13:50:15.518368 29186 ts_tablet_manager.cc:1939] T 11115ffcbd1d472bbe9bb184727ad577 P 337c0ed1b3e648a691bb711be0a84b82: Deleting consensus metadata
I20260927 13:50:15.519603 28596 catalog_manager.cc:4841] TS 337c0ed1b3e648a691bb711be0a84b82 (127.20.133.66:40613): tablet 11115ffcbd1d472bbe9bb184727ad577 (table test-table [id=6addd083ba4f4a3ba4ff90f5b4c34858]) successfully deleted
Sep 27 13:50:15 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.133.65@KRBTEST.COM
I20260927 13:50:15.531803 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:15.500671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.20.133.65:36575)
0927 13:50:15.500955 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:15.500964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:15.508758 (+ 7794us) server_negotiation.cc:407] Connection header received
0927 13:50:15.509356 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:15.509363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:15.509464 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:15.509542 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:15.511428 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.512383 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.513604 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.514056 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.514175 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:15.524529 (+ 10354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:15.524553 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:15.524574 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:15.524621 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:15.528216 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:15.529094 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:15.529102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:15.529105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:15.529268 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:15.529959 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:15.529962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:15.529964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:15.530469 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:15.530777 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:15.531150 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 13:50:15.531421 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 13:50:15.532706 28813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:15.702138 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 28682
I20260927 13:50:15.722644 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 28822
Sep 27 13:50:15 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.133.66@KRBTEST.COM
I20260927 13:50:15.741611 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:15.723706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.20.133.66:40613)
0927 13:50:15.724287 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:15.724293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:15.724403 (+ 110us) server_negotiation.cc:407] Connection header received
0927 13:50:15.724494 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:15.724500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:15.724590 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:15.724722 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:15.726099 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.726974 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.728436 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.728893 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.729023 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:15.733871 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:15.733896 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:15.733919 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:15.733978 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:15.737889 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:15.738941 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:15.738951 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:15.738955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:15.739131 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:15.739590 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:15.739594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:15.739596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:15.740049 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:15.740434 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:15.741012 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 13:50:15.741238 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":253,"threads_started":1}
I20260927 13:50:15.742414 28952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:15.850658 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 28822
I20260927 13:50:15.868593 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 28961
Sep 27 13:50:15 dist-test-slave-q25r krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.133.67@KRBTEST.COM
I20260927 13:50:15.885735 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:15.869477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.20.133.67:44015)
0927 13:50:15.869942 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:15.869946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:15.869967 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:50:15.870029 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:15.870033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:15.870108 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:15.870222 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:15.871692 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.872681 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.874089 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:15.874576 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:15.874709 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:15.879463 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:15.879482 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:15.879500 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:15.879565 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:15.882713 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:15.883508 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:15.883514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:15.883517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:15.883689 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:15.884068 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:15.884072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:15.884075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:15.884522 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:15.884790 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:15.885225 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 13:50:15.885421 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":260,"threads_started":1}
I20260927 13:50:15.886622 29091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:16.034734 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 28961
I20260927 13:50:16.055773 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 28579
I20260927 13:50:16.057263 28651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:16.176208 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 28579
2026-09-27T13:50:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6075 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:50:16 dist-test-slave-q25r krb5kdc[29224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:50:16 dist-test-slave-q25r krb5kdc[29224](info): set up 2 sockets
Sep 27 13:50:16 dist-test-slave-q25r krb5kdc[29224](info): commencing operation
krb5kdc: starting...
W20260927 13:50:17.221261 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:50:17 dist-test-slave-q25r krb5kdc[29224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:50:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:50:17.480026 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:33103
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:43699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:33103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:17.811723 29237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:17.812033 29237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:17.812145 29237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:17.819839 29237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:50:17.819999 29237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:17.820662 29237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:17.820768 29237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:50:17.820842 29237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:50:17.831748 29237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:43699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:33103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:33103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:17.833211 29237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:17.834643 29237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:17.847306 29243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:17.847218 29242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:17.847966 29245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:17.848754 29237 server_base.cc:1034] running on GCE node
I20260927 13:50:17.849808 29237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:17.851658 29237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:17.852929 29237 hybrid_clock.cc:648] HybridClock initialized: now 1790517017852868 us; error 49 us; skew 500 ppm
I20260927 13:50:17.853397 29237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T13:50:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1744 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:50:18 dist-test-slave-q25r krb5kdc[29251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:50:18 dist-test-slave-q25r krb5kdc[29251](info): set up 2 sockets
Sep 27 13:50:18 dist-test-slave-q25r krb5kdc[29251](info): commencing operation
krb5kdc: starting...
W20260927 13:50:18.965145 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:50:19 dist-test-slave-q25r krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:50:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:50:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:50:19.230330 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:38513
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:19.594870 29264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:19.595217 29264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:19.595348 29264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:19.602998 29264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:50:19.603147 29264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:19.603207 29264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:19.603324 29264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:50:19.603375 29264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:50:19.614180 29264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:19.615645 29264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:19.617113 29264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:19.630201 29272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:19.630417 29270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:19.630434 29269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:19.631620 29264 server_base.cc:1034] running on GCE node
I20260927 13:50:19.632737 29264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:19.634591 29264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:19.635836 29264 hybrid_clock.cc:648] HybridClock initialized: now 1790517019635760 us; error 64 us; skew 500 ppm
I20260927 13:50:19.636236 29264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:19 dist-test-slave-q25r krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:19.639884 29264 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:50:19.641767 29264 webserver.cc:460] Webserver started at http://127.20.133.126:34809/ using document root <none> and password file <none>
I20260927 13:50:19.642620 29264 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:19.642729 29264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:19.643062 29264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:19.646663 29264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "c0dd2e8de8eb454999cb5720a504b939"
format_stamp: "Formatted at 2026-09-27 13:50:19 on dist-test-slave-q25r"
server_key: "bb018f15359649062ec3dc4393304577"
server_key_iv: "f6baf6ba8953d51f5f52925fa26c6309"
server_key_version: "encryptionkey@0"
I20260927 13:50:19.647509 29264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "c0dd2e8de8eb454999cb5720a504b939"
format_stamp: "Formatted at 2026-09-27 13:50:19 on dist-test-slave-q25r"
server_key: "bb018f15359649062ec3dc4393304577"
server_key_iv: "f6baf6ba8953d51f5f52925fa26c6309"
server_key_version: "encryptionkey@0"
I20260927 13:50:19.653823 29264 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 13:50:19.658213 29279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:19.660027 29264 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:50:19.660238 29264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "c0dd2e8de8eb454999cb5720a504b939"
format_stamp: "Formatted at 2026-09-27 13:50:19 on dist-test-slave-q25r"
server_key: "bb018f15359649062ec3dc4393304577"
server_key_iv: "f6baf6ba8953d51f5f52925fa26c6309"
server_key_version: "encryptionkey@0"
I20260927 13:50:19.660529 29264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:19.703855 29264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:19.709134 29264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:19.709486 29264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:19.731356 29264 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:38513
I20260927 13:50:19.731360 29330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:38513 every 8 connection(s)
I20260927 13:50:19.733232 29264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:50:19.737401 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 29264
I20260927 13:50:19.737597 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:50:19.737660 29331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:19.738359 21013 external_mini_cluster.cc:1428] Setting key 912ba53f1fbc632c04e9f669b91a6f5d
I20260927 13:50:19.753592 29331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939: Bootstrap starting.
Sep 27 13:50:19 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:50:19.757006 29331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:19.758106 29331 log.cc:824] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:19.762594 29331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939: No bootstrap required, opened a new log
I20260927 13:50:19.768635 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:19.741029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.20.133.126:38513)
0927 13:50:19.742169 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:19.742193 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:19.742261 (+ 68us) server_negotiation.cc:407] Connection header received
0927 13:50:19.744580 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:19.744617 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:19.745251 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:19.745777 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:19.747750 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:19.750057 (+ 2307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:19.751969 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:19.753131 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:19.753381 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:19.758073 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:19.758086 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:19.758113 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:19.758179 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:19.762460 (+ 4281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:19.763676 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:19.763686 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:19.763696 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:19.763996 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:19.764597 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:19.764603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:19.764606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:19.765531 (+ 925us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:19.765882 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:19.766509 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 13:50:19.767066 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":534,"threads_started":1}
I20260927 13:50:19.773003 29331 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } }
I20260927 13:50:19.773406 29331 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:19.773485 29331 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0dd2e8de8eb454999cb5720a504b939, State: Initialized, Role: FOLLOWER
I20260927 13:50:19.774047 29331 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } }
I20260927 13:50:19.774287 29331 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:50:19.774423 29331 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:50:19.774573 29331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:19.777099 29331 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } }
I20260927 13:50:19.777635 29331 leader_election.cc:302] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0dd2e8de8eb454999cb5720a504b939; no voters:
I20260927 13:50:19.778259 29331 leader_election.cc:288] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:50:19.778517 29336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:19.779444 29336 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 1 LEADER]: Becoming Leader. State: Replica: c0dd2e8de8eb454999cb5720a504b939, State: Running, Role: LEADER
I20260927 13:50:19.780121 29336 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } }
I20260927 13:50:19.780637 29331 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:50:19.783209 29338 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0dd2e8de8eb454999cb5720a504b939. Latest consensus state: current_term: 1 leader_uuid: "c0dd2e8de8eb454999cb5720a504b939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } } }
I20260927 13:50:19.783700 29338 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:19.784372 29337 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0dd2e8de8eb454999cb5720a504b939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } } }
I20260927 13:50:19.784700 29337 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:19.793094 29343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:50:19.799726 29343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:50:19.811317 29343 catalog_manager.cc:1347] Generated new cluster ID: 28e47688430949b59ce2f4995f52070e
I20260927 13:50:19.811460 29343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:50:19.833781 29343 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:50:19.836138 29343 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:50:19.843729 29343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939: Generated new TSK 0
I20260927 13:50:19.844775 29343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:50:19.921038 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38513
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:20.278225 29358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:20.278551 29358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:20.278687 29358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:20.286926 29358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:20.287127 29358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:20.287293 29358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:50:20.299329 29358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:20.301318 29358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:20.303017 29358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:20.317247 29363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:20.317401 29364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:20.317878 29366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:20.322140 29358 server_base.cc:1034] running on GCE node
I20260927 13:50:20.322888 29358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:20.323988 29358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:20.325201 29358 hybrid_clock.cc:648] HybridClock initialized: now 1790517020325099 us; error 98 us; skew 500 ppm
I20260927 13:50:20.325546 29358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:20 dist-test-slave-q25r krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517020, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:20.328996 29358 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:50:20.330771 29358 webserver.cc:460] Webserver started at http://127.20.133.65:45469/ using document root <none> and password file <none>
I20260927 13:50:20.331560 29358 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:20.331670 29358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:20.331974 29358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:20.335408 29358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "bc6b2702aae44c35b25e64a5f4c245fb"
format_stamp: "Formatted at 2026-09-27 13:50:20 on dist-test-slave-q25r"
server_key: "32ad152b984e98937e8c349fbcddf041"
server_key_iv: "8e32c62610428626d2b415db8ff44be3"
server_key_version: "encryptionkey@0"
I20260927 13:50:20.336438 29358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "bc6b2702aae44c35b25e64a5f4c245fb"
format_stamp: "Formatted at 2026-09-27 13:50:20 on dist-test-slave-q25r"
server_key: "32ad152b984e98937e8c349fbcddf041"
server_key_iv: "8e32c62610428626d2b415db8ff44be3"
server_key_version: "encryptionkey@0"
I20260927 13:50:20.343981 29358 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 13:50:20.348817 29373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:20.350813 29358 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:50:20.351051 29358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "bc6b2702aae44c35b25e64a5f4c245fb"
format_stamp: "Formatted at 2026-09-27 13:50:20 on dist-test-slave-q25r"
server_key: "32ad152b984e98937e8c349fbcddf041"
server_key_iv: "8e32c62610428626d2b415db8ff44be3"
server_key_version: "encryptionkey@0"
I20260927 13:50:20.351290 29358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:20.370301 29358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:20.375507 29358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:20.375818 29358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:20.377017 29358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:20.378880 29358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:20.378969 29358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:20.379094 29358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:20.379158 29358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:20.416245 29358 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:37597
I20260927 13:50:20.416282 29485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:37597 every 8 connection(s)
I20260927 13:50:20.417563 29358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:50:20.425083 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 29358
I20260927 13:50:20.425307 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:50:20.425961 21013 external_mini_cluster.cc:1428] Setting key 18873f01b264b2b954a61eb596f7da6b
Sep 27 13:50:20 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517020, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:50:20.438720 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:20.420079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:41269 (local address 127.20.133.126:38513)
0927 13:50:20.420521 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:20.420525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:20.421427 (+ 902us) server_negotiation.cc:407] Connection header received
0927 13:50:20.423251 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:20.423270 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:20.423357 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:20.423452 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:20.425281 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:20.426071 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:20.427959 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:20.428448 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:20.428551 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:20.432805 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:20.432811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:20.432814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:20.432853 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:20.435413 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:20.436447 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:20.436453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:20.436456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:20.436599 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:20.436993 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:20.436998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:20.437001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:20.437344 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:20.437479 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:20.438165 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 13:50:20.438384 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":233,"threads_started":1}
I20260927 13:50:20.439535 29489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:20.420569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:41269)
0927 13:50:20.421269 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:20.421307 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:20.422975 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:20.423705 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:20.423714 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:20.424239 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:20.425042 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:20.425071 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:20.426250 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:20.426255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:20.427803 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:20.427819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:20.428688 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:20.428696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:20.428979 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:20.429711 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:20.429745 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:20.432526 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:20.435596 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:20.435608 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:20.435629 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:20.436280 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:20.436705 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:20.436708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:20.436711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:20.436893 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:20.437455 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:20.437460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:20.437791 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 13:50:20.438022 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 13:50:20.438427 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":267,"threads_started":1}
I20260927 13:50:20.441010 29486 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38513
I20260927 13:50:20.441337 29486 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:20.442211 29486 heartbeater.cc:511] Master 127.20.133.126:38513 requested a full tablet report, sending...
I20260927 13:50:20.444584 29296 ts_manager.cc:194] Registered new tserver with Master: bc6b2702aae44c35b25e64a5f4c245fb (127.20.133.65:37597)
I20260927 13:50:20.446103 29296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:41269
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:50:20.479861 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38513
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:20.818410 29492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:20.818737 29492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:20.818866 29492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:20.826691 29492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:20.826871 29492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:20.827083 29492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:50:20.837725 29492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:20.839041 29492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:20.840328 29492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:20.855923 29500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:20.856281 29497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:20.855945 29498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:20.857157 29492 server_base.cc:1034] running on GCE node
I20260927 13:50:20.857944 29492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:20.859016 29492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:20.860250 29492 hybrid_clock.cc:648] HybridClock initialized: now 1790517020860198 us; error 53 us; skew 500 ppm
I20260927 13:50:20.860656 29492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:20 dist-test-slave-q25r krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517020, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:20.869822 29492 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:50:20.871595 29492 webserver.cc:460] Webserver started at http://127.20.133.66:40629/ using document root <none> and password file <none>
I20260927 13:50:20.872371 29492 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:20.872509 29492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:20.872821 29492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:20.875931 29492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "d5808a221c7d44a59d5710f9f7d3fdd4"
format_stamp: "Formatted at 2026-09-27 13:50:20 on dist-test-slave-q25r"
server_key: "b07aab2b8178d87c2d2fbf2c7c99d497"
server_key_iv: "8a97e56b70ca737c0366e9b4f6a1dd0f"
server_key_version: "encryptionkey@0"
I20260927 13:50:20.876820 29492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "d5808a221c7d44a59d5710f9f7d3fdd4"
format_stamp: "Formatted at 2026-09-27 13:50:20 on dist-test-slave-q25r"
server_key: "b07aab2b8178d87c2d2fbf2c7c99d497"
server_key_iv: "8a97e56b70ca737c0366e9b4f6a1dd0f"
server_key_version: "encryptionkey@0"
I20260927 13:50:20.882612 29492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:50:20.886332 29507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:20.887806 29492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:50:20.887990 29492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "d5808a221c7d44a59d5710f9f7d3fdd4"
format_stamp: "Formatted at 2026-09-27 13:50:20 on dist-test-slave-q25r"
server_key: "b07aab2b8178d87c2d2fbf2c7c99d497"
server_key_iv: "8a97e56b70ca737c0366e9b4f6a1dd0f"
server_key_version: "encryptionkey@0"
I20260927 13:50:20.888203 29492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:20.910041 29492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:20.914945 29492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:20.915210 29492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:20.916167 29492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:20.917528 29492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:20.917601 29492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:20.917687 29492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:20.917730 29492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:20.948278 29492 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:37697
I20260927 13:50:20.948320 29619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:37697 every 8 connection(s)
I20260927 13:50:20.949569 29492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:50:20.952950 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 29492
I20260927 13:50:20.953140 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:50:20.953641 21013 external_mini_cluster.cc:1428] Setting key 9a508101ab52f2560705950656b3febd
Sep 27 13:50:20 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517020, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:50:20.971060 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:20.952164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:60037 (local address 127.20.133.126:38513)
0927 13:50:20.952731 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:20.952737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:20.954045 (+ 1308us) server_negotiation.cc:407] Connection header received
0927 13:50:20.955853 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:20.955858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:20.955947 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:20.956038 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:20.957943 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:20.958661 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:20.960226 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:20.960706 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:20.960818 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:20.964648 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:20.964654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:20.964657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:20.964699 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:20.967812 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:20.968694 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:20.968699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:20.968703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:20.968861 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:20.969255 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:20.969260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:20.969263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:20.969596 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:20.969743 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:20.970429 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 13:50:20.970657 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
I20260927 13:50:20.971951 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:20.952762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:60037)
0927 13:50:20.953587 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:20.953634 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:20.955576 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:20.956377 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:20.956391 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:20.956988 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:20.957732 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:20.957754 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:20.958812 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:20.958817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:20.960074 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:20.960087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:20.960939 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:20.960947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:20.961179 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:20.961868 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:20.961897 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:20.964324 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:20.967996 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:20.968005 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:20.968020 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:20.968566 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:20.968956 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:20.968959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:20.968962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:20.969151 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:20.969718 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:20.969724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:20.970063 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 13:50:20.970307 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 13:50:20.970707 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":331,"threads_started":1}
I20260927 13:50:20.973616 29620 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38513
I20260927 13:50:20.973898 29620 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
I20260927 13:50:20.974601 29620 heartbeater.cc:511] Master 127.20.133.126:38513 requested a full tablet report, sending...
I20260927 13:50:20.976078 29296 ts_manager.cc:194] Registered new tserver with Master: d5808a221c7d44a59d5710f9f7d3fdd4 (127.20.133.66:37697)
I20260927 13:50:20.976979 29296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:60037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:50:21.003587 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38513
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:21.336257 29626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:21.336598 29626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:21.336719 29626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:21.344452 29626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:21.344583 29626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:21.344739 29626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:50:21.355340 29626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:21.356706 29626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:21.357964 29626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:21.371274 29634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:21.371207 29631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:21.371208 29632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:21.372196 29626 server_base.cc:1034] running on GCE node
I20260927 13:50:21.373052 29626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:21.374092 29626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:21.375300 29626 hybrid_clock.cc:648] HybridClock initialized: now 1790517021375233 us; error 56 us; skew 500 ppm
I20260927 13:50:21.375661 29626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:21 dist-test-slave-q25r krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:21.378907 29626 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:50:21.380476 29626 webserver.cc:460] Webserver started at http://127.20.133.67:37149/ using document root <none> and password file <none>
I20260927 13:50:21.381217 29626 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:21.381333 29626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:21.381632 29626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:21.384582 29626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "273508e0bb0941e597f1b45affe3fe42"
format_stamp: "Formatted at 2026-09-27 13:50:21 on dist-test-slave-q25r"
server_key: "d3ddfc177d193c337c5c52532592108e"
server_key_iv: "dc1efeaff3a29ae4862bc115f7846f71"
server_key_version: "encryptionkey@0"
I20260927 13:50:21.385370 29626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "273508e0bb0941e597f1b45affe3fe42"
format_stamp: "Formatted at 2026-09-27 13:50:21 on dist-test-slave-q25r"
server_key: "d3ddfc177d193c337c5c52532592108e"
server_key_iv: "dc1efeaff3a29ae4862bc115f7846f71"
server_key_version: "encryptionkey@0"
I20260927 13:50:21.390765 29626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:50:21.395282 29641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:21.397190 29626 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:50:21.397397 29626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "273508e0bb0941e597f1b45affe3fe42"
format_stamp: "Formatted at 2026-09-27 13:50:21 on dist-test-slave-q25r"
server_key: "d3ddfc177d193c337c5c52532592108e"
server_key_iv: "dc1efeaff3a29ae4862bc115f7846f71"
server_key_version: "encryptionkey@0"
I20260927 13:50:21.397629 29626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:21.417915 29626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:21.422364 29626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:21.422652 29626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:21.423637 29626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:21.425061 29626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:21.425143 29626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:21.425266 29626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:21.425318 29626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:21.449788 29486 heartbeater.cc:503] Master 127.20.133.126:38513 was elected leader, sending a full tablet report...
I20260927 13:50:21.459445 29626 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:45545
I20260927 13:50:21.459522 29753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:45545 every 8 connection(s)
I20260927 13:50:21.460860 29626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:50:21.466769 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 29626
I20260927 13:50:21.466958 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:50:21.467473 21013 external_mini_cluster.cc:1428] Setting key f9f7d63d57331619567678790fb83aa4
Sep 27 13:50:21 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:50:21.482525 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:21.464183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:53949 (local address 127.20.133.126:38513)
0927 13:50:21.464436 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:21.464440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:21.465844 (+ 1404us) server_negotiation.cc:407] Connection header received
0927 13:50:21.467933 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:21.467940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:21.468041 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:21.468132 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:21.469850 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:21.470643 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:21.472224 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:21.472686 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:21.472799 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:21.476650 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:21.476655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:21.476658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:21.476695 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:21.479175 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:21.479976 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:21.479980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:21.479983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:21.480112 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:21.480646 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:21.480650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:21.480652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:21.480950 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:21.481113 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:21.481923 (+ 810us) server_negotiation.cc:299] Negotiation successful
0927 13:50:21.482176 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:50:21.483670 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:21.464738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:53949)
0927 13:50:21.465635 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:21.465687 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:21.467585 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:21.468464 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:21.468474 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:21.469045 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:21.469643 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:21.469661 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:21.470806 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:21.470812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:21.472077 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:21.472090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:21.472935 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:21.472942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:21.473175 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:21.473913 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:21.473943 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:21.476389 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:21.479344 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:21.479356 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:21.479374 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:21.479837 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:21.480249 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:21.480254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:21.480256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:21.480520 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:21.481100 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:21.481108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:21.481522 (+ 414us) client_negotiation.cc:769] Sending connection context
0927 13:50:21.481782 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 13:50:21.482366 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":374,"threads_started":1}
I20260927 13:50:21.485631 29754 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38513
I20260927 13:50:21.486122 29754 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:21.487239 29754 heartbeater.cc:511] Master 127.20.133.126:38513 requested a full tablet report, sending...
I20260927 13:50:21.489321 29296 ts_manager.cc:194] Registered new tserver with Master: 273508e0bb0941e597f1b45affe3fe42 (127.20.133.67:45545)
I20260927 13:50:21.490198 29296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:53949
I20260927 13:50:21.496294 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab.
Entry for principal oryx/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab.
WARNING: no policy specified for oryx/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab.
Entry for principal oryx/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab.
WARNING: no policy specified for oryx/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab.
Entry for principal oryx/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab.
I20260927 13:50:21.655766 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 29358
Sep 27 13:50:21 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:50:21.675630 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:21.657177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.20.133.65:37597)
0927 13:50:21.657896 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:21.657906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:21.658359 (+ 453us) server_negotiation.cc:407] Connection header received
0927 13:50:21.658520 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:21.658529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:21.658856 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:21.659048 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:21.660670 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:21.661824 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:21.663462 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:21.664112 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:21.664231 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:21.668952 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:21.668966 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:21.668986 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:21.669035 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:21.672149 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:21.673098 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:21.673107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:21.673110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:21.673274 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:21.673831 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:21.673835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:21.673837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:21.674275 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:21.674561 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:21.674995 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 13:50:21.675273 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":294,"threads_started":1}
I20260927 13:50:21.677201 29481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:21.762818 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 29358
I20260927 13:50:21.777293 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 29492
Sep 27 13:50:21 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:50:21.797047 29766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:21.778470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.20.133.66:37697)
0927 13:50:21.779148 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:21.779157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:21.779186 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:50:21.779257 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:21.779263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:21.779488 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:21.779645 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:21.780991 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:21.782054 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:21.783319 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:21.783869 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:21.783990 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:21.788840 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:21.788873 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:21.788894 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:21.788959 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:21.792981 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:21.794219 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:21.794229 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:21.794232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:21.794490 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:21.795021 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:21.795024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:21.795027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:21.795547 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:21.795837 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:21.796398 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 13:50:21.796690 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":326,"threads_started":1}
I20260927 13:50:21.798563 29615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:21.890769 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 29492
I20260927 13:50:21.904292 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 29626
Sep 27 13:50:21 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:50:21.921890 29768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:21.905315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.20.133.67:45545)
0927 13:50:21.905852 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:21.905861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:21.905895 (+ 34us) server_negotiation.cc:407] Connection header received
0927 13:50:21.905966 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:21.905972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:21.906203 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:21.906354 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:21.907785 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:21.908907 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:21.910300 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:21.910888 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:21.911006 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:21.915780 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:21.915794 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:21.915815 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:21.915860 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:21.918948 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:21.919777 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:21.919786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:21.919788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:21.919938 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:21.920362 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:21.920366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:21.920369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:21.920759 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:21.921011 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:21.921355 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 13:50:21.921583 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":226,"threads_started":1}
I20260927 13:50:21.923177 29749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:22.013540 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 29626
I20260927 13:50:22.027581 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 29264
I20260927 13:50:22.029165 29326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:22.148121 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 29264
I20260927 13:50:22.170926 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:38513
--webserver_interface=127.20.133.126
--webserver_port=34809
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:22.516407 29771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:22.516744 29771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:22.516865 29771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:22.524370 29771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:50:22.524518 29771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:22.524591 29771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:22.524672 29771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:50:22.524737 29771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:50:22.534920 29771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=34809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:22.536278 29771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:22.537582 29771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:22.550470 29776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:22.550402 29777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:22.550364 29779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:22.551771 29771 server_base.cc:1034] running on GCE node
I20260927 13:50:22.552700 29771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:22.554225 29771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:22.555449 29771 hybrid_clock.cc:648] HybridClock initialized: now 1790517022555382 us; error 56 us; skew 500 ppm
I20260927 13:50:22.555860 29771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:22 dist-test-slave-q25r krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517022, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:22.559525 29771 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:50:22.561290 29771 webserver.cc:460] Webserver started at http://127.20.133.126:34809/ using document root <none> and password file <none>
I20260927 13:50:22.562036 29771 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:22.562191 29771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:22.568965 29771 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:50:22.572728 29786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:22.574518 29771 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:50:22.574790 29771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "c0dd2e8de8eb454999cb5720a504b939"
format_stamp: "Formatted at 2026-09-27 13:50:19 on dist-test-slave-q25r"
server_key: "bb018f15359649062ec3dc4393304577"
server_key_iv: "f6baf6ba8953d51f5f52925fa26c6309"
server_key_version: "encryptionkey@0"
I20260927 13:50:22.575867 29771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:22.593106 29771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:22.597355 29771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:22.597620 29771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:22.617343 29771 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:38513
I20260927 13:50:22.617369 29837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:38513 every 8 connection(s)
I20260927 13:50:22.619082 29771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:50:22.627377 29838 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:50:22.628762 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 29771
I20260927 13:50:22.629814 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:37597
--local_ip_for_outbound_sockets=127.20.133.65
--tserver_master_addrs=127.20.133.126:38513
--webserver_port=45469
--webserver_interface=127.20.133.65
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.133.65
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:50:22.632484 29838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939: Bootstrap starting.
I20260927 13:50:22.656323 29838 log.cc:824] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:22.666901 29838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:50:22.667472 29838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939: Bootstrap complete.
I20260927 13:50:22.677821 29838 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } }
I20260927 13:50:22.678825 29838 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0dd2e8de8eb454999cb5720a504b939, State: Initialized, Role: FOLLOWER
I20260927 13:50:22.679651 29838 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } }
I20260927 13:50:22.679934 29838 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:50:22.680055 29838 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:50:22.680229 29838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:50:22.684970 29838 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } }
I20260927 13:50:22.685580 29838 leader_election.cc:302] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0dd2e8de8eb454999cb5720a504b939; no voters:
I20260927 13:50:22.686250 29838 leader_election.cc:288] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:50:22.686410 29842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:50:22.687903 29842 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [term 2 LEADER]: Becoming Leader. State: Replica: c0dd2e8de8eb454999cb5720a504b939, State: Running, Role: LEADER
I20260927 13:50:22.688740 29842 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } }
I20260927 13:50:22.689119 29838 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:50:22.692027 29844 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0dd2e8de8eb454999cb5720a504b939. Latest consensus state: current_term: 2 leader_uuid: "c0dd2e8de8eb454999cb5720a504b939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } } }
I20260927 13:50:22.692545 29844 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:22.692126 29843 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c0dd2e8de8eb454999cb5720a504b939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dd2e8de8eb454999cb5720a504b939" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38513 } } }
I20260927 13:50:22.692704 29843 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:22.702508 29847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:50:22.711193 29847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:50:22.712993 29847 catalog_manager.cc:1259] Loaded cluster ID: 28e47688430949b59ce2f4995f52070e
I20260927 13:50:22.713176 29847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:50:22.717746 29847 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:50:22.719374 29847 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0dd2e8de8eb454999cb5720a504b939: Loaded TSK: 0
I20260927 13:50:22.721422 29847 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 13:50:22.991216 29840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:22.991499 29840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:22.991605 29840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:22.999123 29840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:22.999243 29840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:22.999399 29840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:50:23.010075 29840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.65.keytab
--principal=oryx/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:37597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=45469
--tserver_master_addrs=127.20.133.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:23.011503 29840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:23.012851 29840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:23.029019 29865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:23.029217 29864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:23.029290 29867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:23.031374 29840 server_base.cc:1034] running on GCE node
I20260927 13:50:23.032162 29840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:23.033325 29840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:23.034566 29840 hybrid_clock.cc:648] HybridClock initialized: now 1790517023034497 us; error 64 us; skew 500 ppm
I20260927 13:50:23.034909 29840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517023, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:23.038571 29840 init.cc:377] Logged in from keytab as oryx/127.20.133.65@KRBTEST.COM (short username oryx)
I20260927 13:50:23.040412 29840 webserver.cc:460] Webserver started at http://127.20.133.65:45469/ using document root <none> and password file <none>
I20260927 13:50:23.041172 29840 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:23.041313 29840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:23.047405 29840 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:50:23.050661 29874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:23.052214 29840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:50:23.052459 29840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "bc6b2702aae44c35b25e64a5f4c245fb"
format_stamp: "Formatted at 2026-09-27 13:50:20 on dist-test-slave-q25r"
server_key: "32ad152b984e98937e8c349fbcddf041"
server_key_iv: "8e32c62610428626d2b415db8ff44be3"
server_key_version: "encryptionkey@0"
I20260927 13:50:23.053546 29840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:23.068678 29840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:23.073596 29840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:23.073890 29840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:23.074991 29840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:23.077170 29840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:23.077275 29840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:23.077404 29840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:23.077483 29840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:23.114147 29840 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:37597
I20260927 13:50:23.114235 29986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:37597 every 8 connection(s)
I20260927 13:50:23.115777 29840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:50:23.122426 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 29840
I20260927 13:50:23.123157 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:37697
--local_ip_for_outbound_sockets=127.20.133.66
--tserver_master_addrs=127.20.133.126:38513
--webserver_port=40629
--webserver_interface=127.20.133.66
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.133.66
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:23.138252 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:23.119546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:54749)
0927 13:50:23.120632 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:23.120687 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:23.122748 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:23.124488 (+ 1740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:23.124500 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:23.125226 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:23.125819 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.125837 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.127519 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.127524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.128987 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.129004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.130575 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.130582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.130878 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.131754 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:23.131794 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:23.136753 (+ 4959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":433,"threads_started":1}
W20260927 13:50:23.138281 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:23.119537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:54749 (local address 127.20.133.126:38513)
0927 13:50:23.120555 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:23.120577 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:23.120841 (+ 264us) server_negotiation.cc:407] Connection header received
0927 13:50:23.123468 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:23.123484 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:23.123903 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:23.124274 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:23.126022 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.127349 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.129261 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.130237 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.130469 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.137256 (+ 6787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:54749: BlockingRecv error: recv got EOF from 127.20.133.65:54749 (error 108)
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":304,"threads_started":1}
W20260927 13:50:23.139396 29987 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:38513 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.133.126:38513: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:23.148691 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:23.140240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:34141)
0927 13:50:23.140466 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:23.140486 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:23.140624 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:23.140938 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:23.140943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:23.141335 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:23.141497 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.141506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.142570 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.142574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.143519 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.143533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.144216 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.144222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.144416 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.144987 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:23.145004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:23.148294 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 13:50:23.148844 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:23.140416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:34141 (local address 127.20.133.126:38513)
0927 13:50:23.140634 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:23.140640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:23.140661 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:50:23.140734 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:23.140739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:23.140807 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:23.140893 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:23.141653 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.142417 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.143668 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.144073 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.144184 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.148607 (+ 4423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:34141: BlockingRecv error: recv got EOF from 127.20.133.65:34141 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:23.157917 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:23.149578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:44237)
0927 13:50:23.150052 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:23.150070 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:23.150183 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:23.150497 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:23.150502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:23.150820 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:23.150968 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.150977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.151965 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.151971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.152917 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.152929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.153594 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.153600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.153790 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.154329 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:23.154347 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:23.157551 (+ 3204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364}
W20260927 13:50:23.158422 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:23.149874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:44237 (local address 127.20.133.126:38513)
0927 13:50:23.150040 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:23.150046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:23.150130 (+ 84us) server_negotiation.cc:407] Connection header received
0927 13:50:23.150296 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:23.150300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:23.150374 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:23.150458 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:23.151117 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.151828 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.153050 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.153457 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.153574 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.157906 (+ 4332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:44237: BlockingRecv error: recv got EOF from 127.20.133.65:44237 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 13:50:23.158428 29987 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 13:50:23.517333 29991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:23.517686 29991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:23.517844 29991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:23.525511 29991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:23.525668 29991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:23.525839 29991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:50:23.536041 29991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.66.keytab
--principal=oryx/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:37697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=40629
--tserver_master_addrs=127.20.133.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:23.537492 29991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:23.538901 29991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:23.553131 29996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:23.553103 29997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:23.553196 29999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:23.554045 29991 server_base.cc:1034] running on GCE node
I20260927 13:50:23.554723 29991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:23.555748 29991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:23.556945 29991 hybrid_clock.cc:648] HybridClock initialized: now 1790517023556886 us; error 58 us; skew 500 ppm
I20260927 13:50:23.557303 29991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517023, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:23.560904 29991 init.cc:377] Logged in from keytab as oryx/127.20.133.66@KRBTEST.COM (short username oryx)
I20260927 13:50:23.562847 29991 webserver.cc:460] Webserver started at http://127.20.133.66:40629/ using document root <none> and password file <none>
I20260927 13:50:23.563669 29991 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:23.563802 29991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:23.571094 29991 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 13:50:23.574709 30006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:23.577117 29991 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260927 13:50:23.577318 29991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "d5808a221c7d44a59d5710f9f7d3fdd4"
format_stamp: "Formatted at 2026-09-27 13:50:20 on dist-test-slave-q25r"
server_key: "b07aab2b8178d87c2d2fbf2c7c99d497"
server_key_iv: "8a97e56b70ca737c0366e9b4f6a1dd0f"
server_key_version: "encryptionkey@0"
I20260927 13:50:23.578373 29991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:23.598122 29991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:23.602382 29991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:23.602659 29991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:23.603842 29991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:23.605293 29991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:23.605383 29991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:23.605472 29991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:23.605533 29991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:23.638228 29991 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:37697
I20260927 13:50:23.638286 30118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:37697 every 8 connection(s)
I20260927 13:50:23.639712 29991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:50:23.641913 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 29991
I20260927 13:50:23.642536 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:45545
--local_ip_for_outbound_sockets=127.20.133.67
--tserver_master_addrs=127.20.133.126:38513
--webserver_port=37149
--webserver_interface=127.20.133.67
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.133.67
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:50:23.658711 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:23.642510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:47715 (local address 127.20.133.126:38513)
0927 13:50:23.642716 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:23.642721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:23.644115 (+ 1394us) server_negotiation.cc:407] Connection header received
0927 13:50:23.646095 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:23.646100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:23.646192 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:23.646308 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:23.647915 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.648912 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.651171 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.651832 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.651967 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.658375 (+ 6408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:47715: BlockingRecv error: recv got EOF from 127.20.133.66:47715 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:50:23.659420 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:23.643065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:47715)
0927 13:50:23.643916 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:23.643969 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:23.645784 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:23.646523 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:23.646533 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:23.647110 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:23.647711 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.647730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.649157 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.649165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.650961 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.650977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.652191 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.652202 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.652512 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.653254 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:23.653280 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:23.658138 (+ 4858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":362,"threads_started":1}
W20260927 13:50:23.660066 30119 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:38513 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.133.126:38513: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:23.668480 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:23.660725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:38549)
0927 13:50:23.660914 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:23.660934 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:23.661046 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:23.661356 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:23.661362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:23.661741 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:23.661890 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.661899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.662899 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.662903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.663801 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.663813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.664497 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.664504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.664679 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.665273 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:23.665289 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:23.668141 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 13:50:23.668802 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:23.660800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:38549 (local address 127.20.133.126:38513)
0927 13:50:23.660951 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:23.660956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:23.660991 (+ 35us) server_negotiation.cc:407] Connection header received
0927 13:50:23.661168 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:23.661172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:23.661235 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:23.661315 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:23.662030 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.662759 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.663932 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.664325 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.664475 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.668483 (+ 4008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:38549: BlockingRecv error: recv got EOF from 127.20.133.66:38549 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:23 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:23.678838 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:23.669241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:42435)
0927 13:50:23.669445 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:23.669464 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:23.669586 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:23.669973 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:23.669979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:23.670303 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:23.670467 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.670476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.671507 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.671513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.672721 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:23.672750 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.673672 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.673679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:23.673874 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.674574 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:23.674596 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:23.678534 (+ 3938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 13:50:23.679092 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:23.669500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:42435 (local address 127.20.133.126:38513)
0927 13:50:23.669692 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:23.669698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:23.669717 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:50:23.669775 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:23.669779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:23.669840 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:23.669945 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:23.670615 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.671359 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.673050 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:23.673491 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:23.673611 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:23.678869 (+ 5258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:42435: BlockingRecv error: recv got EOF from 127.20.133.66:42435 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 13:50:23.679296 30119 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 13:50:24.011782 30122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:24.012123 30122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:24.012230 30122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:24.019610 30122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:24.019737 30122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:24.019850 30122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:50:24.030030 30122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/krb5kdc/oryx_127.20.133.67.keytab
--principal=oryx/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:45545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=37149
--tserver_master_addrs=127.20.133.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:24.031332 30122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:24.032617 30122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:24.045536 30128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:24.045490 30127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:24.045569 30130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:24.046298 30122 server_base.cc:1034] running on GCE node
I20260927 13:50:24.046936 30122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:24.047919 30122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:24.049135 30122 hybrid_clock.cc:648] HybridClock initialized: now 1790517024049075 us; error 50 us; skew 500 ppm
I20260927 13:50:24.049453 30122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:24 dist-test-slave-q25r krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517024, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:24.052685 30122 init.cc:377] Logged in from keytab as oryx/127.20.133.67@KRBTEST.COM (short username oryx)
I20260927 13:50:24.054255 30122 webserver.cc:460] Webserver started at http://127.20.133.67:37149/ using document root <none> and password file <none>
I20260927 13:50:24.054960 30122 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:24.055090 30122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:24.060686 30122 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:50:24.063416 30137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:24.064896 30122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:50:24.065065 30122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "273508e0bb0941e597f1b45affe3fe42"
format_stamp: "Formatted at 2026-09-27 13:50:21 on dist-test-slave-q25r"
server_key: "d3ddfc177d193c337c5c52532592108e"
server_key_iv: "dc1efeaff3a29ae4862bc115f7846f71"
server_key_version: "encryptionkey@0"
I20260927 13:50:24.066026 30122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:24.079422 30122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:24.083967 30122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:24.084205 30122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:24.085248 30122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:24.086875 30122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:24.086970 30122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:24.087060 30122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:24.087121 30122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:24.121886 30122 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:45545
I20260927 13:50:24.121992 30249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:45545 every 8 connection(s)
I20260927 13:50:24.123363 30122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:50:24.127807 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 30122
Sep 27 13:50:24 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:24 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:50:24.140650 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:24.126049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:57717 (local address 127.20.133.126:38513)
0927 13:50:24.126236 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:24.126241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:24.127291 (+ 1050us) server_negotiation.cc:407] Connection header received
0927 13:50:24.129777 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:24.129783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:24.129902 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:24.130018 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:24.131550 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.132727 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.134422 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.134929 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.135026 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:24.140364 (+ 5338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:57717: BlockingRecv error: recv got EOF from 127.20.133.67:57717 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:50:24.141510 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:24.126389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:57717)
0927 13:50:24.127110 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:24.127151 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:24.129438 (+ 2287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:24.130223 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:24.130231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:24.130795 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:24.131391 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:24.131408 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.132887 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.132893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:24.134257 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:24.134275 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.135165 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.135172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:24.135459 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:24.136202 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:24.136225 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:24.140155 (+ 3930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":291,"threads_started":1}
W20260927 13:50:24.142256 30250 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:38513 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.133.126:38513: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
I20260927 13:50:24.148067 30254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:24.128922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.20.133.126:38513)
0927 13:50:24.132740 (+ 3818us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:24.132744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:24.132763 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:50:24.132812 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:24.132817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:24.132888 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:24.133002 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:24.134493 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.135446 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.136890 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.137503 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.137595 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:24.140278 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:24.140289 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:24.140312 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:24.140383 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:24.144221 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:24.145040 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:24.145049 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:24.145058 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:24.145284 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:24.145831 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:24.145838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:24.145841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:24.146433 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:24.146799 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:24.147245 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 13:50:24.147614 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3708,"spinlock_wait_cycles":5376,"thread_start_us":3557,"threads_started":1}
Sep 27 13:50:24 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:24 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:24.151724 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:24.142955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:33525)
0927 13:50:24.143161 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:24.143180 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:24.143291 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:24.143654 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:24.143658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:24.143979 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:24.144130 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:24.144138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.145164 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.145168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:24.146092 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:24.146105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.147134 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.147141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:24.147323 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:24.148292 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:24.148314 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:24.151495 (+ 3181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 13:50:24.152369 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:24.143129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:33525 (local address 127.20.133.126:38513)
0927 13:50:24.143329 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:24.143335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:24.143353 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:50:24.143415 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:24.143419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:24.143490 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:24.143593 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:24.144274 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.145038 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.146234 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.146964 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.147099 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:24.151768 (+ 4669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:33525: BlockingRecv error: recv got EOF from 127.20.133.67:33525 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 13:50:24 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:24 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:24.160597 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:24.152412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:58985)
0927 13:50:24.152643 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:24.152657 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:24.152736 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:24.153034 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:24.153038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:24.153317 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:24.153467 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:24.153476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.154560 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.154564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:24.155536 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:24.155551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.156296 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.156301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:24.156514 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:24.157086 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:24.157104 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:24.160323 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134}
W20260927 13:50:24.160893 29989 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:24.152519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:58985 (local address 127.20.133.126:38513)
0927 13:50:24.152721 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:24.152725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:24.152741 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:50:24.152835 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:24.152838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:24.152895 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:24.152963 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:24.153643 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.154377 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.155700 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.156125 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.156229 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:24.160632 (+ 4403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:58985: BlockingRecv error: recv got EOF from 127.20.133.67:58985 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260927 13:50:24.160996 30250 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 13:50:24 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:24 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:24.167757 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:24.159022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:57349)
0927 13:50:24.159510 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:24.159524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:24.159655 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:24.160166 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:24.160170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:24.160783 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:24.161041 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:24.161058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.162074 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.162079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:24.163149 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:24.163163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.164073 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.164080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:24.164238 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:24.164830 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:24.164847 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:24.167424 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":245,"threads_started":1}
W20260927 13:50:24.167917 30254 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:24.159146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:57349 (local address 127.20.133.126:38513)
0927 13:50:24.159387 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:24.159393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:24.159767 (+ 374us) server_negotiation.cc:407] Connection header received
0927 13:50:24.159833 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:24.159839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:24.159920 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:24.160021 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:24.161212 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.161903 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.163291 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.163880 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.164001 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:24.167687 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:57349: BlockingRecv error: recv got EOF from 127.20.133.65:57349 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 13:50:24 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:24 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:24.689152 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:24.679899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:43223)
0927 13:50:24.680481 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:24.680500 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:24.680621 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:24.680956 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:24.680960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:24.681503 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:24.681780 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:24.681793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.682896 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.682901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:24.683955 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:24.683967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.684793 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.684798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:24.684952 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:24.685570 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:24.685588 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:24.688798 (+ 3210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":278,"threads_started":1}
W20260927 13:50:24.689407 30257 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:24.679966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:43223 (local address 127.20.133.126:38513)
0927 13:50:24.680542 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:24.680547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:24.680566 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:50:24.680733 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:24.680737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:24.680805 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:24.680907 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:24.681930 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.682718 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.684113 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:24.684615 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:24.684725 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:24.689097 (+ 4372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:43223: BlockingRecv error: recv got EOF from 127.20.133.66:43223 (error 108)
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":329,"threads_started":1}
Sep 27 13:50:25 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:25 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:25.171636 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:25.161746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:49727)
0927 13:50:25.162324 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:25.162340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:25.162483 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:25.162855 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:25.162859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:25.163357 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:25.163554 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:25.163564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.164649 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.164653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:25.165661 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:25.165674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.166452 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.166459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:25.166613 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:25.167187 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:25.167205 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:25.171267 (+ 4062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":329,"threads_started":1}
W20260927 13:50:25.171911 30257 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:25.161857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:49727 (local address 127.20.133.126:38513)
0927 13:50:25.162079 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:25.162085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:25.162433 (+ 348us) server_negotiation.cc:407] Connection header received
0927 13:50:25.162615 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:25.162620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:25.162706 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:25.162801 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:25.163703 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.164508 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.165799 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.166276 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.166385 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:25.171552 (+ 5167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:49727: BlockingRecv error: recv got EOF from 127.20.133.67:49727 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 13:50:25 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:25 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:25.179621 30259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:25.168713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:53297)
0927 13:50:25.169293 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:25.169331 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:25.169452 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:25.170395 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:25.170403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:25.170917 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:25.171163 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:25.171175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.172773 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.172778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:25.173952 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:25.173973 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.175050 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.175055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:25.175191 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:25.175917 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:25.175940 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:25.179270 (+ 3330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":299,"threads_started":1}
W20260927 13:50:25.179975 30260 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:25.168889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:53297 (local address 127.20.133.126:38513)
0927 13:50:25.169941 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:25.169947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:25.169966 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:50:25.170017 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:25.170022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:25.170097 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:25.170236 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:25.171370 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.172608 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.174158 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.174874 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.174987 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:25.179566 (+ 4579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:53297: BlockingRecv error: recv got EOF from 127.20.133.65:53297 (error 108)
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":725,"threads_started":1}
Sep 27 13:50:25 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:25 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:25.700096 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:25.690489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:48905)
0927 13:50:25.691185 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:25.691210 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:25.691368 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:25.691764 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:25.691768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:25.692210 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:25.692480 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:25.692492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.693602 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.693607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:25.694549 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:25.694562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.695490 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.695498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:25.695671 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:25.696478 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:25.696500 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:25.699790 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":334,"threads_started":1}
W20260927 13:50:25.700404 30262 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:25.690565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:48905 (local address 127.20.133.126:38513)
0927 13:50:25.691399 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:25.691404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:25.691422 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:50:25.691487 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:25.691491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:25.691563 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:25.691667 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:25.692637 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.693441 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.694702 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:25.695285 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:25.695412 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:25.700103 (+ 4691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:48905: BlockingRecv error: recv got EOF from 127.20.133.66:48905 (error 108)
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":612,"threads_started":1}
Sep 27 13:50:26 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:26 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:26.188149 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:26.173019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:34871)
0927 13:50:26.173716 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:26.173737 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:26.173908 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:26.174279 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:26.174284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:26.174741 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:26.174918 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:26.174929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.176025 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.176030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:26.176947 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:26.176961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.177692 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.177698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:26.177852 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:26.183954 (+ 6102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:26.183980 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:26.187765 (+ 3785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":312,"threads_started":1}
Sep 27 13:50:26 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:50:26.188266 30262 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:26.173305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:34871 (local address 127.20.133.126:38513)
0927 13:50:26.173619 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:26.173626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:26.173835 (+ 209us) server_negotiation.cc:407] Connection header received
0927 13:50:26.174056 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:26.174060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:26.174133 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:26.174224 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:26.175086 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.175876 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.177090 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.177533 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.177643 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:26.188045 (+ 10402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:34871: BlockingRecv error: recv got EOF from 127.20.133.67:34871 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 13:50:26 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:26.190327 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:26.180639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:59273)
0927 13:50:26.181226 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:26.181244 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:26.181351 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:26.182075 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:26.182081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:26.182470 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:26.182662 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:26.182672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.183719 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.183725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:26.185113 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:26.185132 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.186057 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.186064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:26.186254 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:26.186877 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:26.186895 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:26.190071 (+ 3176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":300,"threads_started":1}
W20260927 13:50:26.190788 30265 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:26.180703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:59273 (local address 127.20.133.126:38513)
0927 13:50:26.181794 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:26.181800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:26.181816 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:50:26.181855 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:26.181859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:26.181914 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:26.182009 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:26.182824 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.183591 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.185307 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.185884 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.185994 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:26.190439 (+ 4445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:59273: BlockingRecv error: recv got EOF from 127.20.133.65:59273 (error 108)
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":798,"threads_started":1}
Sep 27 13:50:26 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:26 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:26.709108 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:26.701089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:60275)
0927 13:50:26.701578 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:26.701594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:26.701696 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:26.701970 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:26.701973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:26.702325 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:26.702501 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:26.702510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.703548 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.703553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:26.704575 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:26.704591 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.705301 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.705306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:26.705430 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:26.706041 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:26.706059 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:26.708823 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":253,"threads_started":1}
W20260927 13:50:26.709420 30267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:26.701176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:60275 (local address 127.20.133.126:38513)
0927 13:50:26.701613 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:26.701617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:26.701635 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:50:26.701789 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:26.701792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:26.701856 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:26.701942 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:26.702673 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.703379 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.704726 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:26.705162 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:26.705257 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:26.709086 (+ 3829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:60275: BlockingRecv error: recv got EOF from 127.20.133.66:60275 (error 108)
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":250,"threads_started":1}
Sep 27 13:50:27 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:27 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:27 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:27.199174 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:27.189331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:57481)
0927 13:50:27.189860 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:27.189882 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:27.190031 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:27.190484 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:27.190490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:27.190944 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:27.191160 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:27.191173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.192607 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.192613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:27.193725 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:27.193738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.194666 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.194698 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:27.194936 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:27.195699 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:27.195723 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:27.198801 (+ 3078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":828,"threads_started":1}
W20260927 13:50:27.199209 30267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:27.189419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:57481 (local address 127.20.133.126:38513)
0927 13:50:27.189639 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:27.189646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:27.189985 (+ 339us) server_negotiation.cc:407] Connection header received
0927 13:50:27.190224 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:27.190230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:27.190322 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:27.190436 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:27.191327 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.192437 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.193870 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.194497 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.194606 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:27.199018 (+ 4412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:57481: BlockingRecv error: recv got EOF from 127.20.133.67:57481 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 13:50:27 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:27.200706 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:27.191306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:43143)
0927 13:50:27.192004 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:27.192024 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:27.192134 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:27.192449 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:27.192452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:27.192934 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:27.193122 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:27.193133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.194393 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.194398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:27.195804 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:27.195820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.196530 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.196537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:27.196708 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:27.197332 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:27.197356 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:27.200396 (+ 3040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":495,"threads_started":1}
W20260927 13:50:27.200839 30270 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:27.191319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:43143 (local address 127.20.133.126:38513)
0927 13:50:27.191929 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:27.191935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:27.192100 (+ 165us) server_negotiation.cc:407] Connection header received
0927 13:50:27.192220 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:27.192225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:27.192293 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:27.192422 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:27.193289 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.194211 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.195949 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.196392 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.196484 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:27.200598 (+ 4114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:43143: BlockingRecv error: recv got EOF from 127.20.133.65:43143 (error 108)
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":257,"threads_started":1}
Sep 27 13:50:27 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:27 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:27.719458 30271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:27.710219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:46207)
0927 13:50:27.711160 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:27.711181 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:27.711342 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:27.711690 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:27.711696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:27.712155 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:27.712328 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:27.712369 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.713425 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.713431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:27.714441 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:27.714453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.715151 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.715157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:27.715328 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:27.715943 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:27.715962 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:27.719107 (+ 3145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":616,"threads_started":1}
W20260927 13:50:27.719678 30272 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:27.710302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:46207 (local address 127.20.133.126:38513)
0927 13:50:27.710843 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:27.710851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:27.711266 (+ 415us) server_negotiation.cc:407] Connection header received
0927 13:50:27.711465 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:27.711471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:27.711546 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:27.711642 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:27.712500 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.713284 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.714575 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:27.715010 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:27.715121 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:27.719410 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:46207: BlockingRecv error: recv got EOF from 127.20.133.66:46207 (error 108)
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":261,"threads_started":1}
Sep 27 13:50:28 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:28 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:28 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:28.211215 30273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:28.200275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:48827)
0927 13:50:28.200880 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:28.200899 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:28.201067 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:28.201468 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:28.201473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:28.201946 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:28.202181 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:28.202194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.203934 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.203940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:28.205296 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:28.205313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.206096 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.206100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:28.206247 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:28.207009 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:28.207034 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:28.210960 (+ 3926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":320,"threads_started":1}
W20260927 13:50:28.211616 30272 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:28.200450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:48827 (local address 127.20.133.126:38513)
0927 13:50:28.200694 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:28.200701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:28.201003 (+ 302us) server_negotiation.cc:407] Connection header received
0927 13:50:28.201202 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:28.201207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:28.201278 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:28.201375 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:28.202399 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.203751 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.205477 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.205943 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.206074 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:28.211301 (+ 5227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:48827: BlockingRecv error: recv got EOF from 127.20.133.67:48827 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 13:50:28 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:28.212931 30274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:28.201716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:50491)
0927 13:50:28.202328 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:28.202352 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:28.202465 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:28.203868 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:28.203875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:28.204455 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:28.204649 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:28.204659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.205976 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.205983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:28.207245 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:28.207261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.208118 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.208126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:28.208385 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:28.209023 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:28.209052 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:28.212581 (+ 3529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":289,"threads_started":1}
W20260927 13:50:28.213181 30275 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:28.201856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:50491 (local address 127.20.133.126:38513)
0927 13:50:28.203510 (+ 1654us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:28.203514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:28.203533 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:50:28.203579 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:28.203582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:28.203653 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:28.203760 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:28.204811 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.205819 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.207394 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.207884 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.208022 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:28.212878 (+ 4856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:50491: BlockingRecv error: recv got EOF from 127.20.133.65:50491 (error 108)
Metrics: {"server-negotiator.queue_time_us":1533,"thread_start_us":1352,"threads_started":1}
Sep 27 13:50:28 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:28 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:28.730394 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:28.720769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:51727)
0927 13:50:28.721333 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:28.721352 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:28.721510 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:28.721850 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:28.721854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:28.722319 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:28.722546 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:28.722555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.723661 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.723665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:28.724764 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:28.724778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.725529 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.725533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:28.725654 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:28.726352 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:28.726371 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:28.730030 (+ 3659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":246,"threads_started":1}
W20260927 13:50:28.730561 30277 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:28.721038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:51727 (local address 127.20.133.126:38513)
0927 13:50:28.721558 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:28.721563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:28.721582 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:50:28.721638 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:28.721642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:28.721711 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:28.721810 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:28.722720 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.723492 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.724926 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:28.725383 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:28.725501 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:28.730280 (+ 4779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:51727: BlockingRecv error: recv got EOF from 127.20.133.66:51727 (error 108)
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":294,"threads_started":1}
Sep 27 13:50:29 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:29 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:29 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:29.224289 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:29.212963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:56257)
0927 13:50:29.213648 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:29.213669 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:29.213838 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:29.214233 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:29.214238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:29.214854 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:29.215105 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:29.215120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.216807 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.216846 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:29.218415 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:29.218433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.219604 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.219613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:29.219848 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:29.220561 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:29.220583 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:29.223873 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":348,"threads_started":1}
W20260927 13:50:29.224429 30277 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:29.213114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:56257 (local address 127.20.133.126:38513)
0927 13:50:29.213322 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:29.213327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:29.213794 (+ 467us) server_negotiation.cc:407] Connection header received
0927 13:50:29.213981 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:29.213986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:29.214078 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:29.214184 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:29.215261 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.216590 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.218612 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.219366 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.219541 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:29.224155 (+ 4614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:56257: BlockingRecv error: recv got EOF from 127.20.133.67:56257 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 13:50:29 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:29.225607 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:29.214163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:41303)
0927 13:50:29.214810 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:29.214832 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:29.214964 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:29.215688 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:29.215694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:29.216320 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:29.216554 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:29.216566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.218676 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.218682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:29.220035 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:29.220048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.220880 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.220887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:29.221067 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:29.221729 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:29.221749 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:29.225357 (+ 3608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":350,"threads_started":1}
W20260927 13:50:29.225888 30280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:29.214223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:41303 (local address 127.20.133.126:38513)
0927 13:50:29.215225 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:29.215232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:29.215259 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:50:29.215347 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:29.215353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:29.215467 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:29.215657 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:29.216993 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.218509 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.220184 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.220717 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.220840 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:29.225630 (+ 4790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:41303: BlockingRecv error: recv got EOF from 127.20.133.65:41303 (error 108)
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":1024,"threads_started":1}
Sep 27 13:50:29 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:29 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:29.741581 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:29.731489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:42461)
0927 13:50:29.732149 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:29.732170 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:29.732365 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:29.732776 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:29.732781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:29.733299 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:29.733499 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:29.733510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.734567 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.734571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:29.735581 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:29.735596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.736612 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.736618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:29.736768 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:29.737515 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:29.737536 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:29.741217 (+ 3681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":334,"threads_started":1}
W20260927 13:50:29.741865 30282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:29.731596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:42461 (local address 127.20.133.126:38513)
0927 13:50:29.732148 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:29.732152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:29.732272 (+ 120us) server_negotiation.cc:407] Connection header received
0927 13:50:29.732497 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:29.732501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:29.732591 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:29.732691 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:29.733676 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.734423 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.735804 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:29.736395 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:29.736549 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:29.741562 (+ 5013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:42461: BlockingRecv error: recv got EOF from 127.20.133.66:42461 (error 108)
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":338,"threads_started":1}
Sep 27 13:50:30 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:30 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:30.234231 30283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:30.225605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:38659)
0927 13:50:30.226219 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:30.226237 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:30.226385 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:30.226733 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:30.226738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:30.227164 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:30.227334 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:30.227344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.228446 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.228452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:30.229383 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:30.229396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.230170 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.230176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:30.230338 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:30.230940 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:30.230957 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:30.233935 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":305,"threads_started":1}
W20260927 13:50:30.234870 30282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:30.225743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:38659 (local address 127.20.133.126:38513)
0927 13:50:30.225968 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:30.225973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:30.226336 (+ 363us) server_negotiation.cc:407] Connection header received
0927 13:50:30.226507 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:30.226512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:30.226604 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:30.226704 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:30.227483 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.228285 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.229529 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.230010 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.230128 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:30.234188 (+ 4060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:38659: BlockingRecv error: recv got EOF from 127.20.133.67:38659 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 13:50:30 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:30 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:30.246104 30285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:30.226646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:49703)
0927 13:50:30.236829 (+ 10183us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:30.236853 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:30.237022 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:30.237659 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:30.237664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:30.238374 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:30.238628 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:30.238643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.239727 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.239734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:30.240825 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:30.240840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.241582 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.241587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:30.241755 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:30.242529 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:30.242550 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:30.245730 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":10034,"thread_start_us":9783,"threads_started":1}
W20260927 13:50:30.246294 30282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:30.226754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:49703 (local address 127.20.133.126:38513)
0927 13:50:30.235153 (+ 8399us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:30.235157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:30.237009 (+ 1852us) server_negotiation.cc:407] Connection header received
0927 13:50:30.237193 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:30.237199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:30.237286 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:30.237400 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:30.238787 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.239563 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.240980 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.241423 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.241535 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:30.245996 (+ 4461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:49703: BlockingRecv error: recv got EOF from 127.20.133.65:49703 (error 108)
Metrics: {"server-negotiator.queue_time_us":8262,"thread_start_us":10532,"threads_started":1}
Sep 27 13:50:30 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:30 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:30.753136 30286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:30.742922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:57193)
0927 13:50:30.743658 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:30.743676 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:30.743815 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:30.744255 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:30.744260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:30.744858 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:30.745071 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:30.745081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.746257 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.746262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:30.747229 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:30.747241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.748072 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.748077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:30.748208 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:30.748992 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:30.749016 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:30.752678 (+ 3662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":367,"threads_started":1}
W20260927 13:50:30.753211 30282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:30.743127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:57193 (local address 127.20.133.126:38513)
0927 13:50:30.743417 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:30.743422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:30.743798 (+ 376us) server_negotiation.cc:407] Connection header received
0927 13:50:30.743940 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:30.743945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:30.744035 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:30.744160 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:30.745236 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.746073 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.747387 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:30.747877 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:30.747989 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:30.752960 (+ 4971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:57193: BlockingRecv error: recv got EOF from 127.20.133.66:57193 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 13:50:31 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:31 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:31.248703 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:31.235669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:41681)
0927 13:50:31.236316 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:31.236361 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:31.236492 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:31.236864 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:31.236868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:31.237264 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:31.237439 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:31.237447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.238770 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.238776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:31.239776 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:31.239790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.240799 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.240805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:31.240949 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:31.244876 (+ 3927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:31.244896 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:31.248313 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":305,"threads_started":1}
W20260927 13:50:31.249194 30282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:31.235833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:41681 (local address 127.20.133.126:38513)
0927 13:50:31.236068 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:31.236074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:31.236461 (+ 387us) server_negotiation.cc:407] Connection header received
0927 13:50:31.236615 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:31.236620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:31.236714 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:31.236836 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:31.237611 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.238580 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.239941 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.240614 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.240775 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:31.248664 (+ 7889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:41681: BlockingRecv error: recv got EOF from 127.20.133.67:41681 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 13:50:31 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:31 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:31.256812 30288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:31.247492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:56729)
0927 13:50:31.248313 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:31.248362 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:31.248504 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:31.249093 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:31.249098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:31.249566 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:31.249727 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:31.249736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.250931 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.250935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:31.251921 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:31.251933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.252787 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.252792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:31.252917 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:31.253534 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:31.253558 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:31.256466 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":401,"threads_started":1}
W20260927 13:50:31.257220 30289 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:31.247759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:56729 (local address 127.20.133.126:38513)
0927 13:50:31.248616 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:31.248623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:31.248646 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:50:31.248718 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:31.248723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:31.248817 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:31.248959 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:31.249896 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.250774 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.252070 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.252593 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.252702 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:31.256903 (+ 4201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:56729: BlockingRecv error: recv got EOF from 127.20.133.65:56729 (error 108)
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":1231,"threads_started":1}
Sep 27 13:50:31 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:31 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:31.763752 30290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:31.754321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:37153)
0927 13:50:31.754987 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:31.755008 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:31.755154 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:31.755651 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:31.755655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:31.756064 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:31.756231 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:31.756239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.757766 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.757772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:31.758944 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:31.758961 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.759717 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.759722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:31.759843 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:31.760614 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:31.760635 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:31.763492 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":342,"threads_started":1}
W20260927 13:50:31.763993 30289 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:31.754444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:37153 (local address 127.20.133.126:38513)
0927 13:50:31.754664 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:31.754670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:31.755101 (+ 431us) server_negotiation.cc:407] Connection header received
0927 13:50:31.755354 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:31.755359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:31.755449 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:31.755547 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:31.756428 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.757580 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.759104 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:31.759523 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:31.759621 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:31.763737 (+ 4116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:37153: BlockingRecv error: recv got EOF from 127.20.133.66:37153 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 13:50:32 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:32 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:32.260488 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:32.249895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:42025)
0927 13:50:32.250568 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:32.250587 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:32.250746 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:32.251276 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:32.251280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:32.251779 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:32.251980 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:32.251990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.253207 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.253211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:32.254245 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:32.254260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.255202 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.255207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:32.255347 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:32.256061 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:32.256080 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:32.260153 (+ 4073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":342,"threads_started":1}
W20260927 13:50:32.260838 30289 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:32.249978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:42025 (local address 127.20.133.126:38513)
0927 13:50:32.250196 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:32.250202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:32.250682 (+ 480us) server_negotiation.cc:407] Connection header received
0927 13:50:32.250892 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:32.250897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:32.250976 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:32.251102 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:32.252154 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.253013 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.254422 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.254999 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.255125 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:32.260506 (+ 5381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:42025: BlockingRecv error: recv got EOF from 127.20.133.67:42025 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 13:50:32 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:32 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:32.267483 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:32.257836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:54137)
0927 13:50:32.258425 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:32.258447 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:32.258584 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:32.259091 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:32.259096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:32.259555 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:32.259732 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:32.259742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.261178 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.261183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:32.262344 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:32.262357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.263140 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.263146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:32.263378 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:32.264095 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:32.264115 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:32.267197 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":335,"threads_started":1}
W20260927 13:50:32.267782 30293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:32.257935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:54137 (local address 127.20.133.126:38513)
0927 13:50:32.258798 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:32.258803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:32.258818 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:50:32.258872 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:32.258875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:32.258935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:32.259028 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:32.259874 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.261052 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.262471 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.262966 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.263088 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:32.267456 (+ 4368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:54137: BlockingRecv error: recv got EOF from 127.20.133.65:54137 (error 108)
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":586,"threads_started":1}
Sep 27 13:50:32 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:32 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:32.774201 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:32.764923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:43555)
0927 13:50:32.765473 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:32.765489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:32.765635 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:32.765971 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:32.765976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:32.766478 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:32.766659 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:32.766669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.767766 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.767771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:32.768931 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:32.768943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.769701 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.769707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:32.769873 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:32.770616 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:32.770636 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:32.773851 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":292,"threads_started":1}
W20260927 13:50:32.774376 30293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:32.765202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:43555 (local address 127.20.133.126:38513)
0927 13:50:32.765407 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:32.765412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:32.765609 (+ 197us) server_negotiation.cc:407] Connection header received
0927 13:50:32.765766 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:32.765770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:32.765843 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:32.765928 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:32.766807 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.767599 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.769139 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:32.769549 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:32.769659 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:32.774140 (+ 4481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:43555: BlockingRecv error: recv got EOF from 127.20.133.66:43555 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 13:50:33 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:33 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:33.271934 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:33.261719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:38879)
0927 13:50:33.262442 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:33.262460 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:33.262608 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:33.262977 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:33.262981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:33.263480 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:33.263685 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:33.263695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.264855 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.264859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:33.265872 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:33.265884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.266786 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.266794 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:33.266975 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:33.267722 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:33.267748 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:33.271552 (+ 3804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":425,"threads_started":1}
W20260927 13:50:33.272149 30293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:33.261879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:38879 (local address 127.20.133.126:38513)
0927 13:50:33.262106 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:33.262111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:33.262555 (+ 444us) server_negotiation.cc:407] Connection header received
0927 13:50:33.262717 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:33.262722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:33.262797 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:33.262917 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:33.263871 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.264713 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.266044 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.266548 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.266677 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:33.271829 (+ 5152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:38879: BlockingRecv error: recv got EOF from 127.20.133.67:38879 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 13:50:33 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:33 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:33.279171 30296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:33.268688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:42749)
0927 13:50:33.269812 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:33.269838 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:33.269969 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:33.270373 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:33.270379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:33.270877 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:33.271055 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:33.271064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.272866 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.272870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:33.273944 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:33.273958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.274882 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.274887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:33.275024 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:33.275735 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:33.275755 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:33.278895 (+ 3140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":830,"threads_started":1}
W20260927 13:50:33.279580 30297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:33.268715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:42749 (local address 127.20.133.126:38513)
0927 13:50:33.269292 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:33.269298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:33.269935 (+ 637us) server_negotiation.cc:407] Connection header received
0927 13:50:33.270100 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:33.270106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:33.270193 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:33.270301 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:33.271213 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.272718 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.274135 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.274686 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.274830 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:33.279216 (+ 4386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:42749: BlockingRecv error: recv got EOF from 127.20.133.65:42749 (error 108)
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":242,"threads_started":1}
Sep 27 13:50:33 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:33 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:33.787544 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:33.775492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:35833)
0927 13:50:33.776066 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:33.776084 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:33.776230 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:33.776635 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:33.776639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:33.777094 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:33.777278 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:33.777290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.778372 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.778377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:33.779387 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:33.779399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.780407 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.780413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:33.780546 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:33.784478 (+ 3932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:33.784507 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:33.787303 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":370,"threads_started":1}
W20260927 13:50:33.787812 30297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:33.775614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:35833 (local address 127.20.133.126:38513)
0927 13:50:33.775797 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:33.775819 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:33.776197 (+ 378us) server_negotiation.cc:407] Connection header received
0927 13:50:33.776372 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:33.776377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:33.776468 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:33.776590 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:33.777451 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.778229 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.779529 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:33.780166 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:33.780314 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:33.787554 (+ 7240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:35833: BlockingRecv error: recv got EOF from 127.20.133.66:35833 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 13:50:34 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:34 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:34.280962 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:34.272920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:44573)
0927 13:50:34.273406 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:34.273422 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:34.273551 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:34.273878 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:34.273883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:34.274266 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:34.274414 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:34.274424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.275444 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.275449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:34.276452 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:34.276464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.277112 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.277118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:34.277278 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:34.277865 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:34.277882 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:34.280635 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":260,"threads_started":1}
W20260927 13:50:34.281087 30297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:34.273096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:44573 (local address 127.20.133.126:38513)
0927 13:50:34.273300 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:34.273306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:34.273498 (+ 192us) server_negotiation.cc:407] Connection header received
0927 13:50:34.273667 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:34.273672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:34.273747 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:34.273830 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:34.274563 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.275306 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.276577 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.276970 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.277074 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:34.280830 (+ 3756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:44573: BlockingRecv error: recv got EOF from 127.20.133.67:44573 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 13:50:34 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:34 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:34.292393 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:34.280252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:45643)
0927 13:50:34.284925 (+ 4673us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:34.284943 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:34.285057 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:34.285387 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:34.285391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:34.285833 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:34.286021 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:34.286030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.286948 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.286951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:34.287894 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:34.287909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.288718 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.288722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:34.288849 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:34.289422 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:34.289441 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:34.292090 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4559,"thread_start_us":4438,"threads_started":1}
W20260927 13:50:34.292551 30297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:34.280304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:45643 (local address 127.20.133.126:38513)
0927 13:50:34.281307 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:34.281312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:34.285011 (+ 3699us) server_negotiation.cc:407] Connection header received
0927 13:50:34.285140 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:34.285144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:34.285222 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:34.285299 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:34.286140 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.286834 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.288065 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.288562 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.288670 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:34.292299 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:45643: BlockingRecv error: recv got EOF from 127.20.133.65:45643 (error 108)
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":4138,"threads_started":1}
Sep 27 13:50:34 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:34 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:34.797428 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:34.788655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:58921)
0927 13:50:34.789296 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:34.789317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:34.789489 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:34.789916 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:34.789921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:34.790415 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:34.790599 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:34.790608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.791697 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.791700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:34.792734 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:34.792746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.793536 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.793542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:34.793706 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:34.794319 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:34.794336 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:34.797187 (+ 2851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":335,"threads_started":1}
W20260927 13:50:34.797775 30297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:34.788780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:58921 (local address 127.20.133.126:38513)
0927 13:50:34.789007 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:34.789015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:34.789426 (+ 411us) server_negotiation.cc:407] Connection header received
0927 13:50:34.789659 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:34.789665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:34.789761 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:34.789869 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:34.790738 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.791591 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.792898 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:34.793369 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:34.793496 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:34.797494 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:58921: BlockingRecv error: recv got EOF from 127.20.133.66:58921 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 13:50:35 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:35 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:35.291512 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:35.282111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:54969)
0927 13:50:35.282717 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:35.282736 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:35.282943 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:35.283319 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:35.283324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:35.283831 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:35.284057 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:35.284067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.285204 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.285211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:35.286134 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:35.286147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.286831 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.286846 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:35.287024 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:35.287694 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:35.287713 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:35.291160 (+ 3447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":327,"threads_started":1}
W20260927 13:50:35.291797 30297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:35.282213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:54969 (local address 127.20.133.126:38513)
0927 13:50:35.282444 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:35.282451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:35.282859 (+ 408us) server_negotiation.cc:407] Connection header received
0927 13:50:35.283092 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:35.283097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:35.283169 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:35.283270 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:35.284220 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.285068 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.286267 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.286678 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.286785 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:35.291475 (+ 4690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:54969: BlockingRecv error: recv got EOF from 127.20.133.67:54969 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 13:50:35 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:35 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:35.301934 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:35.293331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:55765)
0927 13:50:35.293949 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:35.293967 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:35.294114 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:35.294441 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:35.294446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:35.294921 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:35.295089 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:35.295098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.296140 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.296145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:35.297128 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:35.297139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.297877 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.297883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:35.298059 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:35.298700 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:35.298720 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:35.301556 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":344,"threads_started":1}
W20260927 13:50:35.302080 30297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:35.293368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:55765 (local address 127.20.133.126:38513)
0927 13:50:35.293546 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:35.293554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:35.294051 (+ 497us) server_negotiation.cc:407] Connection header received
0927 13:50:35.294227 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:35.294233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:35.294307 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:35.294396 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:35.295226 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.296002 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.297255 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.297718 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.297825 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:35.301813 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:55765: BlockingRecv error: recv got EOF from 127.20.133.65:55765 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 13:50:35 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:35 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:35.808374 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:35.798607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:56667)
0927 13:50:35.799294 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:35.799314 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:35.799462 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:35.799925 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:35.799929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:35.800447 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:35.800644 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:35.800653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.801653 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.801658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:35.802677 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:35.802688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.803518 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.803522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:35.803662 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:35.804414 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:35.804434 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:35.808070 (+ 3636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":417,"threads_started":1}
W20260927 13:50:35.808583 30297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:35.798761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:56667 (local address 127.20.133.126:38513)
0927 13:50:35.799021 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:35.799026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:35.799431 (+ 405us) server_negotiation.cc:407] Connection header received
0927 13:50:35.799632 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:35.799637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:35.799718 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:35.799840 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:35.800771 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.801499 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.802826 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:35.803365 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:35.803494 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:35.808282 (+ 4788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:56667: BlockingRecv error: recv got EOF from 127.20.133.66:56667 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 27 13:50:36 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:36 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:36.308054 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:36.292838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:38733)
0927 13:50:36.293518 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:36.293537 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:36.293679 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:36.294049 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:36.294052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:36.294512 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:36.294699 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:36.294708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.295862 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.295867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:36.296957 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:36.296975 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.297739 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.297743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:36.297912 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:36.303818 (+ 5906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:36.303884 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:36.307720 (+ 3836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":430,"threads_started":1}
W20260927 13:50:36.308243 30297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:36.292988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:38733 (local address 127.20.133.126:38513)
0927 13:50:36.293222 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:36.293228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:36.293644 (+ 416us) server_negotiation.cc:407] Connection header received
0927 13:50:36.293792 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:36.293797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:36.293890 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:36.294002 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:36.294860 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.295674 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.297135 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.297590 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.297690 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:36.307955 (+ 10265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:38733: BlockingRecv error: recv got EOF from 127.20.133.67:38733 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 13:50:36 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:36 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:36.313475 30307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:36.303032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:42521)
0927 13:50:36.303782 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:36.303806 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:36.303955 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:36.304288 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:36.304293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:36.304832 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:36.305065 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:36.305078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.306497 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.306501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:36.307581 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:36.307594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.308766 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.308774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:36.308983 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:36.309576 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:36.309594 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:36.313093 (+ 3499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":344,"threads_started":1}
W20260927 13:50:36.313714 30308 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:36.303195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:42521 (local address 127.20.133.126:38513)
0927 13:50:36.303991 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:36.303995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:36.304013 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:50:36.304064 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:36.304067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:36.304128 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:36.304224 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:36.305238 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.306320 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.307794 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.308500 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.308643 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:36.313395 (+ 4752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:42521: BlockingRecv error: recv got EOF from 127.20.133.65:42521 (error 108)
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":541,"threads_started":1}
Sep 27 13:50:36 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:36 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:36.818002 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:36.809441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:35177)
0927 13:50:36.810075 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:36.810090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:36.810219 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:36.810777 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:36.810784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:36.811262 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:36.811436 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:36.811445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.812433 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.812438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:36.813373 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:36.813385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.814120 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.814124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:36.814239 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:36.814823 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:36.814843 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:36.817721 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":263,"threads_started":1}
W20260927 13:50:36.818205 30308 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:36.809693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:35177 (local address 127.20.133.126:38513)
0927 13:50:36.809870 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:36.809875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:36.810145 (+ 270us) server_negotiation.cc:407] Connection header received
0927 13:50:36.810555 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:36.810559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:36.810634 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:36.810742 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:36.811579 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.812273 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.813525 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:36.813955 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:36.814050 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:36.817960 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:35177: BlockingRecv error: recv got EOF from 127.20.133.66:35177 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 13:50:37 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:37 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:37.318018 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:37.309407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:32859)
0927 13:50:37.309948 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:37.309965 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:37.310096 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:37.310450 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:37.310454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:37.310862 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:37.311034 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:37.311043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.312017 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.312021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:37.313084 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:37.313101 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.314007 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.314013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:37.314181 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:37.314880 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:37.314907 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:37.317785 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
W20260927 13:50:37.318243 30308 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:37.309523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:32859 (local address 127.20.133.126:38513)
0927 13:50:37.309718 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:37.309722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:37.310092 (+ 370us) server_negotiation.cc:407] Connection header received
0927 13:50:37.310200 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:37.310204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:37.310270 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:37.310362 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:37.311168 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.311886 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.313230 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.313804 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.313918 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:37.317998 (+ 4080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:32859: BlockingRecv error: recv got EOF from 127.20.133.67:32859 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 13:50:37 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:37 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:37.322335 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:37.314493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:40241)
0927 13:50:37.314949 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:37.314969 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:37.315088 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:37.315484 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:37.315488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:37.315860 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:37.316085 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:37.316108 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.317198 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.317204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:37.318328 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:37.318343 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.319007 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.319011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:37.319124 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:37.319602 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:37.319618 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:37.322079 (+ 2461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":233,"threads_started":1}
W20260927 13:50:37.322536 30312 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:37.314533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:40241 (local address 127.20.133.126:38513)
0927 13:50:37.315238 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:37.315243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:37.315258 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:50:37.315293 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:37.315296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:37.315345 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:37.315428 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:37.316249 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.317056 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.318457 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.318889 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.318982 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:37.322285 (+ 3303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:40241: BlockingRecv error: recv got EOF from 127.20.133.65:40241 (error 108)
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":365,"threads_started":1}
Sep 27 13:50:37 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:37 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:37.827790 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:37.819056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:59787)
0927 13:50:37.819619 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:37.819652 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:37.819769 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:37.820111 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:37.820115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:37.820523 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:37.820719 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:37.820729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.821834 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.821839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:37.822797 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:37.822809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.823581 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.823590 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:37.823721 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:37.824324 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:37.824368 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:37.827450 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":354,"threads_started":1}
W20260927 13:50:37.827906 30312 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:37.819223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:59787 (local address 127.20.133.126:38513)
0927 13:50:37.819420 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:37.819425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:37.819749 (+ 324us) server_negotiation.cc:407] Connection header received
0927 13:50:37.819869 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:37.819873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:37.819940 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:37.820029 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:37.820893 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.821668 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.822964 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:37.823401 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:37.823497 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:37.827691 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:59787: BlockingRecv error: recv got EOF from 127.20.133.66:59787 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 13:50:38 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:38 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.67@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:38.328243 30314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:38.319100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.67:35877)
0927 13:50:38.319571 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:38.319590 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:38.319720 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:38.320081 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:38.320085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:38.320611 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:38.320797 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:38.320808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.322206 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.322210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:38.323266 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:38.323280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.324306 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.324312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:38.324518 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:38.325163 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:38.325189 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:38.327939 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":251,"threads_started":1}
W20260927 13:50:38.328461 30312 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:38.319250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:35877 (local address 127.20.133.126:38513)
0927 13:50:38.319451 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:38.319456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:38.319709 (+ 253us) server_negotiation.cc:407] Connection header received
0927 13:50:38.319836 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:38.319840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:38.319918 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:38.320004 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:38.320989 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.322068 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.323413 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.324152 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.324256 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:38.328172 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.67:35877: BlockingRecv error: recv got EOF from 127.20.133.67:35877 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 13:50:38 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:38 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.65@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:38.331878 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:38.323425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.65:60559)
0927 13:50:38.324204 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:38.324222 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:38.324374 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:38.324657 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:38.324661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:38.325238 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:38.325442 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:38.325454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.326466 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.326472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:38.327409 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:38.327423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.328540 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.328547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:38.328699 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:38.329180 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:38.329197 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:38.331624 (+ 2427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":537,"threads_started":1}
W20260927 13:50:38.332062 30316 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:38.323501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:60559 (local address 127.20.133.126:38513)
0927 13:50:38.324049 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:38.324055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:38.324313 (+ 258us) server_negotiation.cc:407] Connection header received
0927 13:50:38.324468 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:38.324472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:38.324522 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:38.324606 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:38.325612 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.326311 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.327552 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.327975 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.328084 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:38.331826 (+ 3742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.65:60559: BlockingRecv error: recv got EOF from 127.20.133.65:60559 (error 108)
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":281,"threads_started":1}
Sep 27 13:50:38 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:50:38 dist-test-slave-q25r krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.133.66@KRBTEST.COM for oryx/127.20.133.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:50:38.837512 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:38.829238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38513 (local address 127.20.133.66:38587)
0927 13:50:38.829806 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:38.829822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:38.829949 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:38.830303 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:38.830309 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:38.830804 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:38.831015 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:38.831028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.832280 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.832286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:38.833247 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:38.833259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.833942 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.833948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:38.834110 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:38.834691 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:38.834709 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:38.837263 (+ 2554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.133.126:38513: Server oryx/127.20.133.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":324,"threads_started":1}
W20260927 13:50:38.837641 30316 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:38.829307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:38587 (local address 127.20.133.126:38513)
0927 13:50:38.829466 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:38.829471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:38.829912 (+ 441us) server_negotiation.cc:407] Connection header received
0927 13:50:38.830066 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:38.830072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:38.830150 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:38.830251 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:38.831175 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.832140 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.833401 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:38.833808 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:38.833916 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:38.837443 (+ 3527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.133.66:38587: BlockingRecv error: recv got EOF from 127.20.133.66:38587 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:50:39.139374 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 29840
I20260927 13:50:39.150905 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:39.140486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.20.133.65:37597)
0927 13:50:39.140983 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:39.140991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:39.141016 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:50:39.141087 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:39.141093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:39.141302 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:39.141450 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:39.142792 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:39.143770 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:39.145034 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:39.145610 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:39.145718 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:39.148120 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:39.148129 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:39.148143 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:39.148184 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:39.150420 (+ 2236us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.133.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 13:50:39.150674 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43130: Request ticket server kudu/127.20.133.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":227,"threads_started":1}
W20260927 13:50:39.151015 30319 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43130: Request ticket server kudu/127.20.133.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 13:50:39.151197 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 29840: Not authorized: Client connection negotiation failed: client connection to 127.20.133.65:37597: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.133.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 13:50:39.151379 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 29840
I20260927 13:50:39.165150 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 29991
I20260927 13:50:39.177296 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:39.166310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.20.133.66:37697)
0927 13:50:39.166944 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:39.166951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:39.166980 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:50:39.167040 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:39.167046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:39.167252 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:39.167413 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:39.168676 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:39.169781 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:39.171055 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:39.171638 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:39.171741 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:39.174194 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:39.174203 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:39.174219 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:39.174261 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:39.176780 (+ 2519us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.133.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 13:50:39.177050 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49596: Request ticket server kudu/127.20.133.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":324,"threads_started":1}
W20260927 13:50:39.177394 30320 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49596: Request ticket server kudu/127.20.133.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 13:50:39.177574 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 29991: Not authorized: Client connection negotiation failed: client connection to 127.20.133.66:37697: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.133.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 13:50:39.177728 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 29991
I20260927 13:50:39.191206 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 30122
I20260927 13:50:39.202590 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:39.192115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.20.133.67:45545)
0927 13:50:39.192676 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:39.192683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:39.192708 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:50:39.192768 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:39.192775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:39.192976 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:39.193124 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:39.194377 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:39.195422 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:39.196626 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:39.197151 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:39.197251 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:39.199626 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:39.199635 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:39.199650 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:39.199681 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:39.202078 (+ 2397us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.133.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 13:50:39.202348 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47552: Request ticket server kudu/127.20.133.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":294,"threads_started":1}
W20260927 13:50:39.202672 30321 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47552: Request ticket server kudu/127.20.133.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 13:50:39.202888 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 30122: Not authorized: Client connection negotiation failed: client connection to 127.20.133.67:45545: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.133.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 13:50:39.203038 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 30122
I20260927 13:50:39.216207 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 29771
I20260927 13:50:39.217494 29833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:39.334789 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 29771
2026-09-27T13:50:39Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21410 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T13:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:50:39Z Disabled control of system clock
I20260927 13:50:39.406392 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:38861
--webserver_interface=127.20.133.126
--webserver_port=0
--builtin_ntp_servers=127.20.133.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:38861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 13:50:39.731230 30326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:39.731536 30326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:39.731639 30326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:39.739058 30326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:50:39.739178 30326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:39.739216 30326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:39.739295 30326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:50:39.739351 30326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:50:39.749428 30326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:38861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:38861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:39.750808 30326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:39.752622 30326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:39.764652 30332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:39.764707 30331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:39.764726 30334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:39.765230 30326 server_base.cc:1034] running on GCE node
I20260927 13:50:39.766120 30326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:39.767710 30326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:39.768929 30326 hybrid_clock.cc:648] HybridClock initialized: now 1790517039768874 us; error 50 us; skew 500 ppm
I20260927 13:50:39.769330 30326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:50:39.771773 30326 webserver.cc:460] Webserver started at http://127.20.133.126:39911/ using document root <none> and password file <none>
I20260927 13:50:39.772584 30326 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:39.772681 30326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:39.772974 30326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:39.775946 30326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "1ce54463eb9c4ab2bdcb08a6c5a6af91"
format_stamp: "Formatted at 2026-09-27 13:50:39 on dist-test-slave-q25r"
server_key: "4c5600e9957e495bd785061c560fb904"
server_key_iv: "6347be6ef1fcd4b707b7329cf21f303d"
server_key_version: "encryptionkey@0"
I20260927 13:50:39.776793 30326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "1ce54463eb9c4ab2bdcb08a6c5a6af91"
format_stamp: "Formatted at 2026-09-27 13:50:39 on dist-test-slave-q25r"
server_key: "4c5600e9957e495bd785061c560fb904"
server_key_iv: "6347be6ef1fcd4b707b7329cf21f303d"
server_key_version: "encryptionkey@0"
I20260927 13:50:39.782075 30326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:50:39.786170 30340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:39.787732 30326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:50:39.787925 30326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "1ce54463eb9c4ab2bdcb08a6c5a6af91"
format_stamp: "Formatted at 2026-09-27 13:50:39 on dist-test-slave-q25r"
server_key: "4c5600e9957e495bd785061c560fb904"
server_key_iv: "6347be6ef1fcd4b707b7329cf21f303d"
server_key_version: "encryptionkey@0"
I20260927 13:50:39.788168 30326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:39.805868 30326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:39.806782 30326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:39.807076 30326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:39.825037 30326 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:38861
I20260927 13:50:39.825057 30391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:38861 every 8 connection(s)
I20260927 13:50:39.826380 30326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:50:39.827771 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 30326
I20260927 13:50:39.827966 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:50:39.828791 21013 external_mini_cluster.cc:1428] Setting key 667c2ac3bf546371fdaf2c367c25932e
I20260927 13:50:39.830812 30392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:39.845171 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:39.831153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.20.133.126:38861)
0927 13:50:39.832395 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:39.832410 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:39.832532 (+ 122us) server_negotiation.cc:407] Connection header received
0927 13:50:39.834363 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:39.834391 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:39.835019 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:39.835478 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:39.836027 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:39.838066 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:39.839454 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:39.840552 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:39.840734 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:39.842045 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:39.842056 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:39.842076 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:39.842845 (+ 769us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:39.842957 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:39.843015 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:39.843166 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:39.843422 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 13:50:39.843804 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":403,"threads_started":1}
I20260927 13:50:39.846050 30392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91: Bootstrap starting.
I20260927 13:50:39.849215 30392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:39.850244 30392 log.cc:824] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:39.853340 30392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91: No bootstrap required, opened a new log
I20260927 13:50:39.861156 30392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce54463eb9c4ab2bdcb08a6c5a6af91" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38861 } }
I20260927 13:50:39.861606 30392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:39.861712 30392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce54463eb9c4ab2bdcb08a6c5a6af91, State: Initialized, Role: FOLLOWER
I20260927 13:50:39.862308 30392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce54463eb9c4ab2bdcb08a6c5a6af91" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38861 } }
I20260927 13:50:39.862524 30392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:50:39.862649 30392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:50:39.862798 30392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:39.865319 30392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce54463eb9c4ab2bdcb08a6c5a6af91" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38861 } }
I20260927 13:50:39.865845 30392 leader_election.cc:302] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ce54463eb9c4ab2bdcb08a6c5a6af91; no voters:
I20260927 13:50:39.866477 30392 leader_election.cc:288] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:50:39.866616 30397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:39.867291 30397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [term 1 LEADER]: Becoming Leader. State: Replica: 1ce54463eb9c4ab2bdcb08a6c5a6af91, State: Running, Role: LEADER
I20260927 13:50:39.867818 30397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce54463eb9c4ab2bdcb08a6c5a6af91" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38861 } }
I20260927 13:50:39.868476 30392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:50:39.870747 30398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ce54463eb9c4ab2bdcb08a6c5a6af91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce54463eb9c4ab2bdcb08a6c5a6af91" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38861 } } }
I20260927 13:50:39.871279 30398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:39.870870 30399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ce54463eb9c4ab2bdcb08a6c5a6af91. Latest consensus state: current_term: 1 leader_uuid: "1ce54463eb9c4ab2bdcb08a6c5a6af91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce54463eb9c4ab2bdcb08a6c5a6af91" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 38861 } } }
I20260927 13:50:39.872248 30405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:50:39.872287 30399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91 [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:39.877627 30405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:50:39.887217 30405 catalog_manager.cc:1347] Generated new cluster ID: d5fa0eb9174142cfbbf1e8fa5849561c
I20260927 13:50:39.887368 30405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:50:39.898615 30405 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:50:39.899906 30405 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:50:39.915213 30405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ce54463eb9c4ab2bdcb08a6c5a6af91: Generated new TSK 0
I20260927 13:50:39.916210 30405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:50:39.930966 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38861
--builtin_ntp_servers=127.20.133.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 13:50:40.269999 30416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:40.270298 30416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:40.270388 30416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:40.277895 30416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:40.278012 30416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:40.278129 30416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:50:40.287518 30416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:40.288864 30416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:40.290683 30416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:40.303773 30421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:40.303839 30424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:40.303817 30422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:40.304672 30416 server_base.cc:1034] running on GCE node
I20260927 13:50:40.305271 30416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:40.306274 30416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:40.307473 30416 hybrid_clock.cc:648] HybridClock initialized: now 1790517040307406 us; error 57 us; skew 500 ppm
I20260927 13:50:40.307789 30416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:50:40.310035 30416 webserver.cc:460] Webserver started at http://127.20.133.65:45559/ using document root <none> and password file <none>
I20260927 13:50:40.310859 30416 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:40.310953 30416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:40.311275 30416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:40.314530 30416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "16dbac2b9e504f0ca6863a52b6bb7233"
format_stamp: "Formatted at 2026-09-27 13:50:40 on dist-test-slave-q25r"
server_key: "328330885577e06721d52842a0b0df7c"
server_key_iv: "91230bf7779d175b6e96b12aaeade670"
server_key_version: "encryptionkey@0"
I20260927 13:50:40.315621 30416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "16dbac2b9e504f0ca6863a52b6bb7233"
format_stamp: "Formatted at 2026-09-27 13:50:40 on dist-test-slave-q25r"
server_key: "328330885577e06721d52842a0b0df7c"
server_key_iv: "91230bf7779d175b6e96b12aaeade670"
server_key_version: "encryptionkey@0"
I20260927 13:50:40.321144 30416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 13:50:40.324882 30430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:40.326314 30416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 13:50:40.326486 30416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "16dbac2b9e504f0ca6863a52b6bb7233"
format_stamp: "Formatted at 2026-09-27 13:50:40 on dist-test-slave-q25r"
server_key: "328330885577e06721d52842a0b0df7c"
server_key_iv: "91230bf7779d175b6e96b12aaeade670"
server_key_version: "encryptionkey@0"
I20260927 13:50:40.326713 30416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:40.347914 30416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:40.348901 30416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:40.349227 30416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:40.350203 30416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:40.351787 30416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:40.351899 30416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:40.352007 30416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:40.352082 30416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:40.385713 30416 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:37857
I20260927 13:50:40.385787 30542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:37857 every 8 connection(s)
I20260927 13:50:40.387068 30416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:50:40.387420 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 30416
I20260927 13:50:40.387621 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:50:40.388471 21013 external_mini_cluster.cc:1428] Setting key 18a91aa27f5dca4d0bff02688a9af556
I20260927 13:50:40.391960 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38861
--builtin_ntp_servers=127.20.133.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 13:50:40.400903 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:40.389899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:55841 (local address 127.20.133.126:38861)
0927 13:50:40.390375 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:40.390382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:40.391768 (+ 1386us) server_negotiation.cc:407] Connection header received
0927 13:50:40.393733 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:40.393739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:40.393828 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:40.393919 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:40.395059 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:40.395771 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:40.397444 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:40.398007 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:40.398155 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:40.399937 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:40.399942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:40.399945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:40.399968 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:40.400012 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:40.400025 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:40.400111 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:40.400440 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 13:50:40.400540 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":223,"threads_started":1}
I20260927 13:50:40.402462 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:40.390591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38861 (local address 127.20.133.65:55841)
0927 13:50:40.391582 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:40.391628 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:40.393444 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:40.394174 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:40.394184 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:40.394279 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:40.394884 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:40.394904 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:40.395928 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:40.395932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:40.397282 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:40.397297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:40.398290 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:40.398299 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:40.398707 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:40.399498 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:50:40.399502 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:40.399560 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:50:40.399680 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:40.400126 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:40.400136 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:40.400145 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 13:50:40.400380 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 13:50:40.400855 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":312,"threads_started":1}
I20260927 13:50:40.404453 30543 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38861
I20260927 13:50:40.405014 30543 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:40.406288 30543 heartbeater.cc:511] Master 127.20.133.126:38861 requested a full tablet report, sending...
I20260927 13:50:40.408763 30357 ts_manager.cc:194] Registered new tserver with Master: 16dbac2b9e504f0ca6863a52b6bb7233 (127.20.133.65:37857)
I20260927 13:50:40.410323 30357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.65:55841
W20260927 13:50:40.734565 30547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:40.734916 30547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:40.735045 30547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:40.742570 30547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:40.742712 30547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:40.742851 30547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:50:40.752552 30547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:40.753898 30547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:40.755709 30547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:40.770277 30552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:40.770264 30553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:40.770717 30555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:40.773989 30547 server_base.cc:1034] running on GCE node
I20260927 13:50:40.774904 30547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:40.776022 30547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:40.777242 30547 hybrid_clock.cc:648] HybridClock initialized: now 1790517040777169 us; error 71 us; skew 500 ppm
I20260927 13:50:40.777542 30547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:50:40.779798 30547 webserver.cc:460] Webserver started at http://127.20.133.66:42837/ using document root <none> and password file <none>
I20260927 13:50:40.780534 30547 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:40.780627 30547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:40.780891 30547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:40.783725 30547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "e134ccbc95f440709a55c72f0b5dde9c"
format_stamp: "Formatted at 2026-09-27 13:50:40 on dist-test-slave-q25r"
server_key: "cc598e3b2e42f94dc5975766fcf7b2f8"
server_key_iv: "1189744ae928958b9105770acccde95f"
server_key_version: "encryptionkey@0"
I20260927 13:50:40.784551 30547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "e134ccbc95f440709a55c72f0b5dde9c"
format_stamp: "Formatted at 2026-09-27 13:50:40 on dist-test-slave-q25r"
server_key: "cc598e3b2e42f94dc5975766fcf7b2f8"
server_key_iv: "1189744ae928958b9105770acccde95f"
server_key_version: "encryptionkey@0"
I20260927 13:50:40.789673 30547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:50:40.793154 30561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:40.794524 30547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:50:40.794679 30547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "e134ccbc95f440709a55c72f0b5dde9c"
format_stamp: "Formatted at 2026-09-27 13:50:40 on dist-test-slave-q25r"
server_key: "cc598e3b2e42f94dc5975766fcf7b2f8"
server_key_iv: "1189744ae928958b9105770acccde95f"
server_key_version: "encryptionkey@0"
I20260927 13:50:40.794927 30547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:40.818779 30547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:40.819715 30547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:40.820015 30547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:40.821045 30547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:40.822573 30547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:40.822662 30547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:40.822769 30547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:40.822834 30547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:40.854951 30547 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:42557
I20260927 13:50:40.855036 30673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:42557 every 8 connection(s)
I20260927 13:50:40.856227 30547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:50:40.857717 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 30547
I20260927 13:50:40.857900 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:50:40.858430 21013 external_mini_cluster.cc:1428] Setting key e673a4110468d367efbd7d4cd6dd98d2
I20260927 13:50:40.861369 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38861
--builtin_ntp_servers=127.20.133.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 13:50:40.870220 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:40.859202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:48319 (local address 127.20.133.126:38861)
0927 13:50:40.859379 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:40.859383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:40.860944 (+ 1561us) server_negotiation.cc:407] Connection header received
0927 13:50:40.863137 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:40.863143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:40.863257 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:40.863347 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:40.864657 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:40.865380 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:40.867075 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:40.867507 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:40.867627 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:40.869363 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:40.869368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:40.869372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:40.869394 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:40.869429 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:40.869442 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:40.869528 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:40.869819 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 13:50:40.869910 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:50:40.871462 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:40.859732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38861 (local address 127.20.133.66:48319)
0927 13:50:40.860792 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:40.860835 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:40.862795 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:40.863616 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:40.863627 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:40.863732 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:40.864477 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:40.864498 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:40.865549 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:40.865553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:40.866904 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:40.866923 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:40.867754 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:40.867762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:40.868127 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:40.868969 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:50:40.868973 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:40.869028 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:50:40.869158 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:40.869533 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:40.869554 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:40.869563 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 13:50:40.869767 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 13:50:40.870218 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":326,"threads_started":1}
I20260927 13:50:40.873047 30674 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38861
I20260927 13:50:40.873387 30674 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:40.874326 30674 heartbeater.cc:511] Master 127.20.133.126:38861 requested a full tablet report, sending...
I20260927 13:50:40.875969 30357 ts_manager.cc:194] Registered new tserver with Master: e134ccbc95f440709a55c72f0b5dde9c (127.20.133.66:42557)
I20260927 13:50:40.876890 30357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.66:48319
W20260927 13:50:41.196959 30677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:41.197273 30677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:41.197386 30677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:41.205331 30677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:41.205456 30677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:41.205603 30677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:50:41.215502 30677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:38861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:41.216868 30677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:41.218672 30677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:41.232101 30685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:41.232177 30682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:41.232314 30683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:41.232997 30677 server_base.cc:1034] running on GCE node
I20260927 13:50:41.233656 30677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:41.234704 30677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:41.235941 30677 hybrid_clock.cc:648] HybridClock initialized: now 1790517041235874 us; error 54 us; skew 500 ppm
I20260927 13:50:41.236287 30677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:50:41.238736 30677 webserver.cc:460] Webserver started at http://127.20.133.67:41811/ using document root <none> and password file <none>
I20260927 13:50:41.239475 30677 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:41.239591 30677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:41.239921 30677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:41.242935 30677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "e2cf44e3b8f14d5a9194a87e467a0d68"
format_stamp: "Formatted at 2026-09-27 13:50:41 on dist-test-slave-q25r"
server_key: "716ff9b9c5ae96e42b99b7ebf52db634"
server_key_iv: "1bb99f74335dfc1b8463688b6d568fed"
server_key_version: "encryptionkey@0"
I20260927 13:50:41.243777 30677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "e2cf44e3b8f14d5a9194a87e467a0d68"
format_stamp: "Formatted at 2026-09-27 13:50:41 on dist-test-slave-q25r"
server_key: "716ff9b9c5ae96e42b99b7ebf52db634"
server_key_iv: "1bb99f74335dfc1b8463688b6d568fed"
server_key_version: "encryptionkey@0"
I20260927 13:50:41.249330 30677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 13:50:41.253005 30691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:41.254482 30677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:50:41.254683 30677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "e2cf44e3b8f14d5a9194a87e467a0d68"
format_stamp: "Formatted at 2026-09-27 13:50:41 on dist-test-slave-q25r"
server_key: "716ff9b9c5ae96e42b99b7ebf52db634"
server_key_iv: "1bb99f74335dfc1b8463688b6d568fed"
server_key_version: "encryptionkey@0"
I20260927 13:50:41.254902 30677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:41.269184 30677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:41.270102 30677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:41.270371 30677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:41.271298 30677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:41.272802 30677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:41.272891 30677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:41.272974 30677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:41.273036 30677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:41.301888 30677 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:43581
I20260927 13:50:41.301936 30803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:43581 every 8 connection(s)
I20260927 13:50:41.303169 30677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:50:41.307052 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 30677
I20260927 13:50:41.307217 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:50:41.307773 21013 external_mini_cluster.cc:1428] Setting key 5b45d393ef84bcce01b39dc1df079c1e
I20260927 13:50:41.323112 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:41.305906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:47521 (local address 127.20.133.126:38861)
0927 13:50:41.306072 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:41.306076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:41.307616 (+ 1540us) server_negotiation.cc:407] Connection header received
0927 13:50:41.309967 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:41.309973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:41.310059 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:41.310165 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:41.311343 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:41.312184 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:41.313930 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:41.314351 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:41.314457 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:41.322203 (+ 7746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:41.322211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:41.322215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:41.322239 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:41.322286 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:41.322306 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:41.322399 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:41.322679 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 13:50:41.322772 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:50:41.324299 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:41.306342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:38861 (local address 127.20.133.67:47521)
0927 13:50:41.307424 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:41.307478 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:41.309652 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:41.310348 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:41.310356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:41.310448 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:41.311175 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:41.311195 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:41.312383 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:41.312392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:41.313770 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:41.313785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:41.314578 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:41.314584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:41.314808 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:41.321815 (+ 7007us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:50:41.321818 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:41.321877 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:50:41.321991 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:41.322425 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:41.322439 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:41.322447 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:50:41.322623 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 13:50:41.323049 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":300,"threads_started":1}
I20260927 13:50:41.325892 30804 heartbeater.cc:347] Connected to a master server at 127.20.133.126:38861
I20260927 13:50:41.326211 30804 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:41.326978 30804 heartbeater.cc:511] Master 127.20.133.126:38861 requested a full tablet report, sending...
I20260927 13:50:41.328630 30357 ts_manager.cc:194] Registered new tserver with Master: e2cf44e3b8f14d5a9194a87e467a0d68 (127.20.133.67:43581)
I20260927 13:50:41.329574 30357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.67:47521
I20260927 13:50:41.336295 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 13:50:41.342267 30545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:41.340838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.20.133.126:38861)
0927 13:50:41.341000 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:41.341005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:41.341414 (+ 409us) server_negotiation.cc:407] Connection header received
0927 13:50:41.341566 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:41.341570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:41.341649 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:41.341737 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:41.342059 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44072: BlockingRecv error: recv got EOF from 127.0.0.1:44072 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:50:41.342737 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 30416
I20260927 13:50:41.350622 30813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:41.343739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.20.133.65:37857)
0927 13:50:41.344210 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:41.344219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:41.344245 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:50:41.344302 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:41.344308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:41.344568 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:41.344704 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:41.345114 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:41.346056 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:41.347320 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:41.347911 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:41.348038 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:41.349144 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:41.349154 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:41.349171 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:41.349792 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:41.349843 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:41.349863 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:41.349965 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:41.350123 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 13:50:41.350276 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":227,"threads_started":1}
I20260927 13:50:41.351783 30538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:41.433534 30543 heartbeater.cc:503] Master 127.20.133.126:38861 was elected leader, sending a full tablet report...
I20260927 13:50:41.434612 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 30416
W20260927 13:50:41.447651 30344 connection.cc:535] server connection from 127.20.133.65:55841 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 13:50:41.447803 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 30547
I20260927 13:50:41.455597 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:41.448668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.20.133.66:42557)
0927 13:50:41.449221 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:41.449228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:41.449257 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:50:41.449318 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:41.449324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:41.449538 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:41.449674 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:41.450071 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:41.451111 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:41.452438 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:41.453071 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:41.453182 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:41.454150 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:41.454158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:41.454173 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:41.454794 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:41.454843 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:41.454863 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:41.454952 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:41.455112 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 13:50:41.455245 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":285,"threads_started":1}
I20260927 13:50:41.456847 30669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:41.538669 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 30547
I20260927 13:50:41.551631 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 30677
I20260927 13:50:41.559943 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:41.552719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.20.133.67:43581)
0927 13:50:41.553365 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:41.553372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:41.553403 (+ 31us) server_negotiation.cc:407] Connection header received
0927 13:50:41.553462 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:41.553468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:41.553682 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:41.553807 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:41.554178 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:41.555234 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:41.556732 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:41.557325 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:41.557426 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:41.558446 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:41.558454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:41.558469 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:41.559094 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:41.559156 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:41.559176 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:41.559279 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:41.559410 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 13:50:41.559546 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":342,"threads_started":1}
I20260927 13:50:41.561321 30799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:41.644029 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 30677
I20260927 13:50:41.656607 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 30326
I20260927 13:50:41.657734 30387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:41.759804 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 30326
2026-09-27T13:50:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2419 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T13:50:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:50:41Z Disabled control of system clock
I20260927 13:50:41.829115 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:45205
--webserver_interface=127.20.133.126
--webserver_port=0
--builtin_ntp_servers=127.20.133.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:45205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 13:50:42.155839 30823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:42.156162 30823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:42.156270 30823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:42.163776 30823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:50:42.163902 30823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:42.163939 30823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:42.164024 30823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:50:42.164098 30823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:50:42.173765 30823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:45205
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:45205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:42.175065 30823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:42.176826 30823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:42.189024 30828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:42.189098 30831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:42.189179 30829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:42.190276 30823 server_base.cc:1034] running on GCE node
I20260927 13:50:42.191155 30823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:42.192644 30823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:42.193864 30823 hybrid_clock.cc:648] HybridClock initialized: now 1790517042193810 us; error 57 us; skew 500 ppm
I20260927 13:50:42.194206 30823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:50:42.196556 30823 webserver.cc:460] Webserver started at http://127.20.133.126:34241/ using document root <none> and password file <none>
I20260927 13:50:42.197253 30823 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:42.197346 30823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:42.197609 30823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:42.200556 30823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "89ca10d0df0949888b3e05811f5008ef"
format_stamp: "Formatted at 2026-09-27 13:50:42 on dist-test-slave-q25r"
server_key: "2fa3f5539044bba274184c7161421c02"
server_key_iv: "b2eb7510f305ad2b9b582c8c3e36e548"
server_key_version: "encryptionkey@0"
I20260927 13:50:42.201331 30823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "89ca10d0df0949888b3e05811f5008ef"
format_stamp: "Formatted at 2026-09-27 13:50:42 on dist-test-slave-q25r"
server_key: "2fa3f5539044bba274184c7161421c02"
server_key_iv: "b2eb7510f305ad2b9b582c8c3e36e548"
server_key_version: "encryptionkey@0"
I20260927 13:50:42.206909 30823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:50:42.210986 30837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:42.212702 30823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:50:42.212903 30823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "89ca10d0df0949888b3e05811f5008ef"
format_stamp: "Formatted at 2026-09-27 13:50:42 on dist-test-slave-q25r"
server_key: "2fa3f5539044bba274184c7161421c02"
server_key_iv: "b2eb7510f305ad2b9b582c8c3e36e548"
server_key_version: "encryptionkey@0"
I20260927 13:50:42.213148 30823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:42.219316 30823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:42.220113 30823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:42.220423 30823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:42.239537 30823 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:45205
I20260927 13:50:42.239621 30888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:45205 every 8 connection(s)
I20260927 13:50:42.241272 30823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:50:42.241743 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 30823
I20260927 13:50:42.241994 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:50:42.242913 21013 external_mini_cluster.cc:1428] Setting key 0589df79ba6e91885e32665b4b683628
I20260927 13:50:42.248050 30889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:42.257616 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:42.248297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.20.133.126:45205)
0927 13:50:42.250301 (+ 2004us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:42.250317 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:42.250383 (+ 66us) server_negotiation.cc:407] Connection header received
0927 13:50:42.252971 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:42.253003 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:42.253694 (+ 691us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:42.254173 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:42.255589 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:42.255596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:42.255611 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:42.255628 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:42.255706 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:42.255724 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:42.255951 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:42.256082 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 13:50:42.256455 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1357,"thread_start_us":968,"threads_started":1}
I20260927 13:50:42.263336 30889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef: Bootstrap starting.
I20260927 13:50:42.266891 30889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:42.268000 30889 log.cc:824] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:42.271623 30889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef: No bootstrap required, opened a new log
I20260927 13:50:42.279417 30889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ca10d0df0949888b3e05811f5008ef" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45205 } }
I20260927 13:50:42.279904 30889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:42.280020 30889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ca10d0df0949888b3e05811f5008ef, State: Initialized, Role: FOLLOWER
I20260927 13:50:42.280709 30889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ca10d0df0949888b3e05811f5008ef" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45205 } }
I20260927 13:50:42.280972 30889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:50:42.281116 30889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:50:42.281278 30889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:42.283622 30889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ca10d0df0949888b3e05811f5008ef" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45205 } }
I20260927 13:50:42.284157 30889 leader_election.cc:302] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ca10d0df0949888b3e05811f5008ef; no voters:
I20260927 13:50:42.284885 30889 leader_election.cc:288] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:50:42.285096 30894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:42.285959 30894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [term 1 LEADER]: Becoming Leader. State: Replica: 89ca10d0df0949888b3e05811f5008ef, State: Running, Role: LEADER
I20260927 13:50:42.286666 30894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ca10d0df0949888b3e05811f5008ef" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45205 } }
I20260927 13:50:42.287300 30889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:50:42.290805 30895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89ca10d0df0949888b3e05811f5008ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ca10d0df0949888b3e05811f5008ef" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45205 } } }
I20260927 13:50:42.291268 30895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:42.290943 30896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89ca10d0df0949888b3e05811f5008ef. Latest consensus state: current_term: 1 leader_uuid: "89ca10d0df0949888b3e05811f5008ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ca10d0df0949888b3e05811f5008ef" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45205 } } }
I20260927 13:50:42.291525 30896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:42.293612 30901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:50:42.300594 30901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:50:42.315172 30901 catalog_manager.cc:1347] Generated new cluster ID: 775ee16244ea4543b505b31cfe2f8216
I20260927 13:50:42.315346 30901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:50:42.333838 30901 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:50:42.334398 30901 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:50:42.349037 30901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89ca10d0df0949888b3e05811f5008ef: Generated new TSK 0
I20260927 13:50:42.350482 30901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:50:42.367173 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45205
--builtin_ntp_servers=127.20.133.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 13:50:42.746272 30913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:42.746640 30913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:42.746754 30913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:42.754997 30913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:42.755154 30913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:42.755334 30913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:50:42.765756 30913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:42.767464 30913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:42.769809 30913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:42.784842 30918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:42.784950 30919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:42.785065 30921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:42.786048 30913 server_base.cc:1034] running on GCE node
I20260927 13:50:42.786692 30913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:42.787837 30913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:42.789067 30913 hybrid_clock.cc:648] HybridClock initialized: now 1790517042788987 us; error 65 us; skew 500 ppm
I20260927 13:50:42.789448 30913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:50:42.791846 30913 webserver.cc:460] Webserver started at http://127.20.133.65:33401/ using document root <none> and password file <none>
I20260927 13:50:42.792654 30913 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:42.792755 30913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:42.793073 30913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:42.797683 30913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "776c7d7dd13a45968de13f11d1c0b875"
format_stamp: "Formatted at 2026-09-27 13:50:42 on dist-test-slave-q25r"
server_key: "a3479d9e5f32382d872837357a337fb5"
server_key_iv: "7fdcf3457495bb6c7b6a8a2b472ec4d5"
server_key_version: "encryptionkey@0"
I20260927 13:50:42.798503 30913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "776c7d7dd13a45968de13f11d1c0b875"
format_stamp: "Formatted at 2026-09-27 13:50:42 on dist-test-slave-q25r"
server_key: "a3479d9e5f32382d872837357a337fb5"
server_key_iv: "7fdcf3457495bb6c7b6a8a2b472ec4d5"
server_key_version: "encryptionkey@0"
I20260927 13:50:42.804247 30913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 13:50:42.808121 30927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:42.809804 30913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 13:50:42.809980 30913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "776c7d7dd13a45968de13f11d1c0b875"
format_stamp: "Formatted at 2026-09-27 13:50:42 on dist-test-slave-q25r"
server_key: "a3479d9e5f32382d872837357a337fb5"
server_key_iv: "7fdcf3457495bb6c7b6a8a2b472ec4d5"
server_key_version: "encryptionkey@0"
I20260927 13:50:42.810192 30913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:42.816548 30913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:42.817471 30913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:42.817718 30913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:42.818706 30913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:42.820191 30913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:42.820282 30913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:42.820403 30913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:42.820466 30913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:42.854007 30913 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:37207
I20260927 13:50:42.854099 31039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:37207 every 8 connection(s)
I20260927 13:50:42.855455 30913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:50:42.856984 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 30913
I20260927 13:50:42.857228 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:50:42.858081 21013 external_mini_cluster.cc:1428] Setting key 896db7b475181207ad021d1f5019559f
I20260927 13:50:42.861568 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45205
--builtin_ntp_servers=127.20.133.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 13:50:42.865803 31042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:42.858979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:44009 (local address 127.20.133.126:45205)
0927 13:50:42.859624 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:42.859632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:42.861160 (+ 1528us) server_negotiation.cc:407] Connection header received
0927 13:50:42.863178 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:42.863184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:42.863278 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:42.863394 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:42.864934 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:42.864941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:42.864946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:42.864968 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:42.865018 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:42.865031 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:42.865155 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:42.865338 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 13:50:42.865431 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":295,"threads_started":1}
I20260927 13:50:42.867048 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:42.859668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45205 (local address 127.20.133.65:44009)
0927 13:50:42.860957 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:42.861012 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:42.862873 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:42.863684 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:42.863695 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:42.863813 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:42.864602 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:50:42.864605 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:42.864653 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:50:42.864756 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:42.865121 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:42.865132 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:42.865140 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:50:42.865436 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 13:50:42.865769 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":360,"threads_started":1}
I20260927 13:50:42.869091 31040 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45205
I20260927 13:50:42.869529 31040 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:42.870368 31040 heartbeater.cc:511] Master 127.20.133.126:45205 requested a full tablet report, sending...
I20260927 13:50:42.873185 30854 ts_manager.cc:194] Registered new tserver with Master: 776c7d7dd13a45968de13f11d1c0b875 (127.20.133.65:37207)
W20260927 13:50:43.217674 31044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:43.217964 31044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:43.218096 31044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:43.225585 31044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:43.225723 31044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:43.225860 31044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:50:43.235206 31044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:43.236634 31044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:43.238497 31044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:43.254035 31050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:43.254160 31049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:43.254920 31044 server_base.cc:1034] running on GCE node
W20260927 13:50:43.254958 31052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:43.255671 31044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:43.256775 31044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:43.257993 31044 hybrid_clock.cc:648] HybridClock initialized: now 1790517043257922 us; error 59 us; skew 500 ppm
I20260927 13:50:43.258343 31044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:50:43.260708 31044 webserver.cc:460] Webserver started at http://127.20.133.66:34339/ using document root <none> and password file <none>
I20260927 13:50:43.261418 31044 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:43.261523 31044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:43.261824 31044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:43.264842 31044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "31e9d06a989e4e5e9d310d9162715889"
format_stamp: "Formatted at 2026-09-27 13:50:43 on dist-test-slave-q25r"
server_key: "e3dd99f3b8e793e4acafd4b9c1ef1ccc"
server_key_iv: "873af2bef6f9ac1bc32e315a4ea08283"
server_key_version: "encryptionkey@0"
I20260927 13:50:43.265682 31044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "31e9d06a989e4e5e9d310d9162715889"
format_stamp: "Formatted at 2026-09-27 13:50:43 on dist-test-slave-q25r"
server_key: "e3dd99f3b8e793e4acafd4b9c1ef1ccc"
server_key_iv: "873af2bef6f9ac1bc32e315a4ea08283"
server_key_version: "encryptionkey@0"
I20260927 13:50:43.271039 31044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 13:50:43.275537 31058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:43.277022 31044 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:50:43.277186 31044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "31e9d06a989e4e5e9d310d9162715889"
format_stamp: "Formatted at 2026-09-27 13:50:43 on dist-test-slave-q25r"
server_key: "e3dd99f3b8e793e4acafd4b9c1ef1ccc"
server_key_iv: "873af2bef6f9ac1bc32e315a4ea08283"
server_key_version: "encryptionkey@0"
I20260927 13:50:43.277410 31044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:43.283598 31044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:43.284534 31044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:43.284807 31044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:43.285797 31044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:43.287292 31044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:43.287377 31044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:43.287549 31044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:43.287643 31044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 13:50:43.318502 31044 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:33603
I20260927 13:50:43.318617 31170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:33603 every 8 connection(s)
I20260927 13:50:43.319795 31044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:50:43.320392 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 31044
I20260927 13:50:43.320605 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:50:43.321163 21013 external_mini_cluster.cc:1428] Setting key c9f7b3d992cdb9ce8685fe93ebc536e6
I20260927 13:50:43.324189 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45205
--builtin_ntp_servers=127.20.133.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 13:50:43.328080 31042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:43.322390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:34511 (local address 127.20.133.126:45205)
0927 13:50:43.322551 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:43.322555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:43.324280 (+ 1725us) server_negotiation.cc:407] Connection header received
0927 13:50:43.326040 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:43.326046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:43.326115 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:43.326216 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:43.327427 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:43.327432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:43.327437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:43.327455 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:43.327493 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:43.327508 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:43.327641 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:43.327753 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 13:50:43.327817 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:50:43.329018 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:43.323032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45205 (local address 127.20.133.66:34511)
0927 13:50:43.324095 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:43.324147 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:43.325774 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:43.326400 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:43.326410 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:43.326505 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:43.327144 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:50:43.327147 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:43.327185 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:50:43.327275 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:43.327592 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:43.327597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:43.327603 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 13:50:43.327808 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 13:50:43.328011 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":325,"threads_started":1}
I20260927 13:50:43.330421 31171 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45205
I20260927 13:50:43.330756 31171 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:43.331439 31171 heartbeater.cc:511] Master 127.20.133.126:45205 requested a full tablet report, sending...
I20260927 13:50:43.333083 30854 ts_manager.cc:194] Registered new tserver with Master: 31e9d06a989e4e5e9d310d9162715889 (127.20.133.66:33603)
W20260927 13:50:43.655611 31174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:43.655949 31174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:43.656056 31174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:43.663771 31174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:43.663918 31174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:43.664103 31174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:50:43.674121 31174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:43.675405 31174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:43.677242 31174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:43.690595 31180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:43.690629 31182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:43.690558 31179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:43.691900 31174 server_base.cc:1034] running on GCE node
I20260927 13:50:43.692624 31174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:43.693674 31174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:43.694912 31174 hybrid_clock.cc:648] HybridClock initialized: now 1790517043694849 us; error 54 us; skew 500 ppm
I20260927 13:50:43.695307 31174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:50:43.697693 31174 webserver.cc:460] Webserver started at http://127.20.133.67:33793/ using document root <none> and password file <none>
I20260927 13:50:43.698436 31174 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:43.698549 31174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:43.698848 31174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:43.701869 31174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "1c60451118ff4932a25c9c42799c61e5"
format_stamp: "Formatted at 2026-09-27 13:50:43 on dist-test-slave-q25r"
server_key: "981fa97f48c9c7d0fe446a9dc962152c"
server_key_iv: "93a6084304795692babaaf32c3c25051"
server_key_version: "encryptionkey@0"
I20260927 13:50:43.702673 31174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "1c60451118ff4932a25c9c42799c61e5"
format_stamp: "Formatted at 2026-09-27 13:50:43 on dist-test-slave-q25r"
server_key: "981fa97f48c9c7d0fe446a9dc962152c"
server_key_iv: "93a6084304795692babaaf32c3c25051"
server_key_version: "encryptionkey@0"
I20260927 13:50:43.707875 31174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 13:50:43.711517 31188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:43.713001 31174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:50:43.713152 31174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "1c60451118ff4932a25c9c42799c61e5"
format_stamp: "Formatted at 2026-09-27 13:50:43 on dist-test-slave-q25r"
server_key: "981fa97f48c9c7d0fe446a9dc962152c"
server_key_iv: "93a6084304795692babaaf32c3c25051"
server_key_version: "encryptionkey@0"
I20260927 13:50:43.713346 31174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:43.719218 31174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:43.719974 31174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:43.720187 31174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:43.721083 31174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:43.722425 31174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:43.722508 31174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:43.722594 31174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:43.722646 31174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:43.751600 31174 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:37441
I20260927 13:50:43.751668 31300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:37441 every 8 connection(s)
I20260927 13:50:43.752928 31174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:50:43.758520 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 31174
I20260927 13:50:43.758685 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:50:43.759101 21013 external_mini_cluster.cc:1428] Setting key b235835562e3edfad46e40b7e3483f06
I20260927 13:50:43.761958 31042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:43.755571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:39381 (local address 127.20.133.126:45205)
0927 13:50:43.755777 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:43.755782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:43.757203 (+ 1421us) server_negotiation.cc:407] Connection header received
0927 13:50:43.758960 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:43.758966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:43.759040 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:43.759143 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:43.760688 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:43.760694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:43.760697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:43.760715 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:43.760750 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:43.760763 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:43.760882 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:43.761122 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 13:50:43.761192 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 13:50:43.763055 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:43.756088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45205 (local address 127.20.133.67:39381)
0927 13:50:43.757040 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:43.757076 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:43.758681 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:43.759394 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:43.759403 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:43.759501 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:43.760157 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:50:43.760162 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:43.760219 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:50:43.760470 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:43.760883 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:43.760896 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:43.760904 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:50:43.761197 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 13:50:43.761923 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":321,"threads_started":1}
I20260927 13:50:43.764964 31301 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45205
I20260927 13:50:43.765306 31301 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:43.765950 31301 heartbeater.cc:511] Master 127.20.133.126:45205 requested a full tablet report, sending...
I20260927 13:50:43.767470 30854 ts_manager.cc:194] Registered new tserver with Master: 1c60451118ff4932a25c9c42799c61e5 (127.20.133.67:37441)
I20260927 13:50:43.775467 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 13:50:43.781706 31042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:50:43.780086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.20.133.126:45205)
0927 13:50:43.780298 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:43.780305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:43.780718 (+ 413us) server_negotiation.cc:407] Connection header received
0927 13:50:43.780891 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:43.780897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:43.780958 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:43.781047 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:43.781395 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33476: BlockingRecv error: recv got EOF from 127.0.0.1:33476 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:50:43.782785 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 30913
I20260927 13:50:43.786327 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:43.783854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.20.133.65:37207)
0927 13:50:43.784381 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:43.784391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:43.784424 (+ 33us) server_negotiation.cc:407] Connection header received
0927 13:50:43.784494 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:43.784500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:43.784731 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:43.784863 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:43.785730 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:43.785736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:43.785748 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:43.785766 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:43.785809 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:43.785823 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:43.785928 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:43.785980 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 13:50:43.786072 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":242,"threads_started":1}
I20260927 13:50:43.787794 31035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:43.870687 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 30913
I20260927 13:50:43.884999 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 31044
I20260927 13:50:43.888619 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:43.886008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.20.133.66:33603)
0927 13:50:43.886501 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:43.886508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:43.886533 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:50:43.886614 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:43.886620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:43.886825 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:43.886959 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:43.887887 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:43.887894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:43.887905 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:43.887923 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:43.887963 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:43.887978 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:43.888095 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:43.888215 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 13:50:43.888319 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":228,"threads_started":1}
I20260927 13:50:43.889843 31166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:43.982057 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 31044
I20260927 13:50:44.004200 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 31174
I20260927 13:50:44.009012 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:44.005562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.20.133.67:37441)
0927 13:50:44.006427 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:44.006435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:44.006474 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:50:44.006551 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:44.006557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:44.006795 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:44.006946 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:44.008150 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:44.008158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:44.008170 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:50:44.008190 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:44.008236 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:50:44.008251 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:44.008476 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:44.008548 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 13:50:44.008646 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":451,"threads_started":1}
I20260927 13:50:44.010632 31296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:44.095196 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 31174
I20260927 13:50:44.110327 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 30823
I20260927 13:50:44.111697 30884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:44.215435 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 30823
2026-09-27T13:50:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2459 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:50:44 dist-test-slave-q25r krb5kdc[31319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:50:44 dist-test-slave-q25r krb5kdc[31319](info): set up 2 sockets
Sep 27 13:50:44 dist-test-slave-q25r krb5kdc[31319](info): commencing operation
krb5kdc: starting...
W20260927 13:50:45.253362 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:50:45 dist-test-slave-q25r krb5kdc[31319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:50:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:50:45.518461 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:45029
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:45029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:45.898890 31332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:45.899302 31332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:45.899446 31332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:45.907531 31332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:50:45.907697 31332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:45.907773 31332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:45.907853 31332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:50:45.907917 31332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:50:45.919303 31332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:45029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:45029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:45.921096 31332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:45.922683 31332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:45.936086 31340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:45.936139 31338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:45.936049 31337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:45.937395 31332 server_base.cc:1034] running on GCE node
I20260927 13:50:45.938330 31332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:45.940058 31332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:45.941272 31332 hybrid_clock.cc:648] HybridClock initialized: now 1790517045941199 us; error 59 us; skew 500 ppm
I20260927 13:50:45.941674 31332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:45 dist-test-slave-q25r krb5kdc[31319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517045, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:45.945264 31332 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:50:45.947122 31332 webserver.cc:460] Webserver started at http://127.20.133.126:44233/ using document root <none> and password file <none>
I20260927 13:50:45.947973 31332 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:45.948091 31332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:45.948491 31332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:45.952507 31332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "b6ecdff4d19447f7a6fcb7e5f37d20cb"
format_stamp: "Formatted at 2026-09-27 13:50:45 on dist-test-slave-q25r"
server_key: "2fc8d705a6051fa2069a305c94a5eef5"
server_key_iv: "7241feafb830277ddabe1d2156ce303e"
server_key_version: "encryptionkey@0"
I20260927 13:50:45.953476 31332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "b6ecdff4d19447f7a6fcb7e5f37d20cb"
format_stamp: "Formatted at 2026-09-27 13:50:45 on dist-test-slave-q25r"
server_key: "2fc8d705a6051fa2069a305c94a5eef5"
server_key_iv: "7241feafb830277ddabe1d2156ce303e"
server_key_version: "encryptionkey@0"
I20260927 13:50:45.960246 31332 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260927 13:50:45.964856 31347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:45.966668 31332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:50:45.966877 31332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "b6ecdff4d19447f7a6fcb7e5f37d20cb"
format_stamp: "Formatted at 2026-09-27 13:50:45 on dist-test-slave-q25r"
server_key: "2fc8d705a6051fa2069a305c94a5eef5"
server_key_iv: "7241feafb830277ddabe1d2156ce303e"
server_key_version: "encryptionkey@0"
I20260927 13:50:45.967123 31332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:46.010756 31332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:46.016511 31332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:46.016839 31332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:46.037031 31332 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:45029
I20260927 13:50:46.037114 31398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:45029 every 8 connection(s)
I20260927 13:50:46.038534 31332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:50:46.043522 31399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:46.047066 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 31332
I20260927 13:50:46.047325 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:50:46.047959 21013 external_mini_cluster.cc:1428] Setting key 05e2fd2f8c2f35882cb01a76be8fc4df
I20260927 13:50:46.059042 31399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb: Bootstrap starting.
I20260927 13:50:46.063423 31399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:46.064764 31399 log.cc:824] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:50:46 dist-test-slave-q25r krb5kdc[31319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:50:46.068822 31399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb: No bootstrap required, opened a new log
I20260927 13:50:46.075778 31402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:46.050478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.20.133.126:45029)
0927 13:50:46.051465 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:46.051485 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:46.051555 (+ 70us) server_negotiation.cc:407] Connection header received
0927 13:50:46.054066 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:46.054105 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:46.054836 (+ 731us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:46.055493 (+ 657us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:46.057317 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:46.059358 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:46.060953 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:46.062053 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:46.062253 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:46.066914 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:46.066926 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:46.066949 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:46.067012 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:46.070570 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:46.071762 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:46.071776 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:46.071783 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:46.072022 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:46.072520 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:46.072524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:46.072526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:46.073227 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:46.073527 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:46.074005 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 13:50:46.074503 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":395,"threads_started":1}
I20260927 13:50:46.079312 31399 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ecdff4d19447f7a6fcb7e5f37d20cb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45029 } }
I20260927 13:50:46.079879 31399 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:46.079985 31399 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ecdff4d19447f7a6fcb7e5f37d20cb, State: Initialized, Role: FOLLOWER
I20260927 13:50:46.080672 31399 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ecdff4d19447f7a6fcb7e5f37d20cb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45029 } }
I20260927 13:50:46.080906 31399 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:50:46.081036 31399 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:50:46.081202 31399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:46.083650 31399 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ecdff4d19447f7a6fcb7e5f37d20cb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45029 } }
I20260927 13:50:46.084262 31399 leader_election.cc:302] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6ecdff4d19447f7a6fcb7e5f37d20cb; no voters:
I20260927 13:50:46.085042 31399 leader_election.cc:288] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:50:46.085333 31404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:46.086769 31404 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [term 1 LEADER]: Becoming Leader. State: Replica: b6ecdff4d19447f7a6fcb7e5f37d20cb, State: Running, Role: LEADER
I20260927 13:50:46.087476 31404 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ecdff4d19447f7a6fcb7e5f37d20cb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45029 } }
I20260927 13:50:46.087853 31399 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:50:46.091722 31405 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6ecdff4d19447f7a6fcb7e5f37d20cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ecdff4d19447f7a6fcb7e5f37d20cb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45029 } } }
I20260927 13:50:46.092216 31405 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:46.091717 31406 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ecdff4d19447f7a6fcb7e5f37d20cb. Latest consensus state: current_term: 1 leader_uuid: "b6ecdff4d19447f7a6fcb7e5f37d20cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ecdff4d19447f7a6fcb7e5f37d20cb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45029 } } }
I20260927 13:50:46.093212 31406 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb [sys.catalog]: This master's current role is: LEADER
I20260927 13:50:46.093398 31411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:50:46.100498 31411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:50:46.111063 31411 catalog_manager.cc:1347] Generated new cluster ID: 5a6053addc2342ae876e819f3531b99d
I20260927 13:50:46.111203 31411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:50:46.123585 31411 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:50:46.125196 31411 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:50:46.135222 31411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb: Generated new TSK 0
I20260927 13:50:46.136255 31411 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:50:46.217067 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45029
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:46.661392 31425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:46.661756 31425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:46.661868 31425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:46.669763 31425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:46.669914 31425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:46.670051 31425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:50:46.681113 31425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:46.682761 31425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:46.684353 31425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:46.699637 31430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:46.699771 31433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:46.699659 31431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:46.700740 31425 server_base.cc:1034] running on GCE node
I20260927 13:50:46.701502 31425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:46.702756 31425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:46.704018 31425 hybrid_clock.cc:648] HybridClock initialized: now 1790517046703951 us; error 52 us; skew 500 ppm
I20260927 13:50:46.704459 31425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:46 dist-test-slave-q25r krb5kdc[31319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517046, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:46.711258 31425 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:50:46.713251 31425 webserver.cc:460] Webserver started at http://127.20.133.65:39639/ using document root <none> and password file <none>
I20260927 13:50:46.714082 31425 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:46.714193 31425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:46.714491 31425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:46.718050 31425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "0928186b6254443297374842b1b0374d"
format_stamp: "Formatted at 2026-09-27 13:50:46 on dist-test-slave-q25r"
server_key: "45eca589056e3d87c4875167c70a2196"
server_key_iv: "47e68a6005df4e5bf98db64e09d4b500"
server_key_version: "encryptionkey@0"
I20260927 13:50:46.718989 31425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "0928186b6254443297374842b1b0374d"
format_stamp: "Formatted at 2026-09-27 13:50:46 on dist-test-slave-q25r"
server_key: "45eca589056e3d87c4875167c70a2196"
server_key_iv: "47e68a6005df4e5bf98db64e09d4b500"
server_key_version: "encryptionkey@0"
I20260927 13:50:46.725610 31425 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 13:50:46.729957 31440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:46.731752 31425 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:50:46.731931 31425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "0928186b6254443297374842b1b0374d"
format_stamp: "Formatted at 2026-09-27 13:50:46 on dist-test-slave-q25r"
server_key: "45eca589056e3d87c4875167c70a2196"
server_key_iv: "47e68a6005df4e5bf98db64e09d4b500"
server_key_version: "encryptionkey@0"
I20260927 13:50:46.732133 31425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:46.766781 31425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:46.772104 31425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:46.772466 31425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:46.773674 31425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:46.775252 31425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:46.775342 31425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:46.775466 31425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:46.775527 31425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:46.812672 31425 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:43029
I20260927 13:50:46.812768 31552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:43029 every 8 connection(s)
I20260927 13:50:46.814258 31425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:50:46.821652 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 31425
I20260927 13:50:46.821897 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:50:46.822669 21013 external_mini_cluster.cc:1428] Setting key 6fc68fa32f4417adeead7b4ded200bbc
Sep 27 13:50:46 dist-test-slave-q25r krb5kdc[31319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517046, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:50:46.838151 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:46.817360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:32949 (local address 127.20.133.126:45029)
0927 13:50:46.817915 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:46.817920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:46.819057 (+ 1137us) server_negotiation.cc:407] Connection header received
0927 13:50:46.821092 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:46.821098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:46.821192 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:46.821331 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:46.823523 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:46.824674 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:46.826856 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:46.827300 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:46.827470 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:46.831626 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:46.831632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:46.831637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:46.831675 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:46.834742 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:46.835552 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:46.835559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:46.835563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:46.835732 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:46.836175 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:46.836180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:46.836184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:46.836604 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:46.836745 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:46.837513 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 13:50:46.837749 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":411,"thread_start_us":274,"threads_started":1}
I20260927 13:50:46.839093 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:46.818044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45029 (local address 127.20.133.65:32949)
0927 13:50:46.818857 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:46.818910 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:46.820779 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:46.821656 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:46.821667 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:46.822438 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:46.823295 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:46.823321 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:46.824846 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:46.824851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:46.826689 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:46.826705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:46.827576 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:46.827582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:46.827851 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:46.828674 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:46.828708 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:46.831371 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:46.834902 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:46.834910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:46.834926 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:46.835423 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:46.835839 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:46.835842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:46.835845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:46.836056 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:46.836741 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:46.836750 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:46.837136 (+ 386us) client_negotiation.cc:769] Sending connection context
0927 13:50:46.837400 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 13:50:46.837880 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":310,"threads_started":1}
I20260927 13:50:46.840937 31553 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45029
I20260927 13:50:46.841421 31553 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:46.842550 31553 heartbeater.cc:511] Master 127.20.133.126:45029 requested a full tablet report, sending...
I20260927 13:50:46.845544 31364 ts_manager.cc:194] Registered new tserver with Master: 0928186b6254443297374842b1b0374d (127.20.133.65:43029)
I20260927 13:50:46.847259 31364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:32949
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:50:46.883682 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45029
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:47.247823 31559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:47.248165 31559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:47.248314 31559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:47.255784 31559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:47.255915 31559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:47.256100 31559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:50:47.266232 31559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:47.267642 31559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:47.269047 31559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:47.284000 31564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:47.284096 31565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:47.284053 31567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:47.285207 31559 server_base.cc:1034] running on GCE node
I20260927 13:50:47.285950 31559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:47.287148 31559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:47.288410 31559 hybrid_clock.cc:648] HybridClock initialized: now 1790517047288329 us; error 69 us; skew 500 ppm
I20260927 13:50:47.288869 31559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:47 dist-test-slave-q25r krb5kdc[31319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517047, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:47.292800 31559 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:50:47.294627 31559 webserver.cc:460] Webserver started at http://127.20.133.66:34761/ using document root <none> and password file <none>
I20260927 13:50:47.295542 31559 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:47.295668 31559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:47.295995 31559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:47.299472 31559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "bdd8aaa37e374822b81d311ccc2110c8"
format_stamp: "Formatted at 2026-09-27 13:50:47 on dist-test-slave-q25r"
server_key: "25bdaaee9b51c21c48b99352b209b48a"
server_key_iv: "220ba31cf069b7a400cd2ecd6c6b2679"
server_key_version: "encryptionkey@0"
I20260927 13:50:47.300318 31559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "bdd8aaa37e374822b81d311ccc2110c8"
format_stamp: "Formatted at 2026-09-27 13:50:47 on dist-test-slave-q25r"
server_key: "25bdaaee9b51c21c48b99352b209b48a"
server_key_iv: "220ba31cf069b7a400cd2ecd6c6b2679"
server_key_version: "encryptionkey@0"
I20260927 13:50:47.306564 31559 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 13:50:47.310936 31574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:47.312697 31559 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:50:47.312888 31559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "bdd8aaa37e374822b81d311ccc2110c8"
format_stamp: "Formatted at 2026-09-27 13:50:47 on dist-test-slave-q25r"
server_key: "25bdaaee9b51c21c48b99352b209b48a"
server_key_iv: "220ba31cf069b7a400cd2ecd6c6b2679"
server_key_version: "encryptionkey@0"
I20260927 13:50:47.313100 31559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:47.347492 31559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:47.352677 31559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:47.352998 31559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:47.354122 31559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:47.355778 31559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:47.355870 31559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:47.356007 31559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:47.356060 31559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:47.391330 31559 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:46747
I20260927 13:50:47.391428 31686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:46747 every 8 connection(s)
I20260927 13:50:47.392904 31559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:50:47.402365 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 31559
I20260927 13:50:47.402586 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:50:47.403056 21013 external_mini_cluster.cc:1428] Setting key 0f9780c4b17be8366293b97898239ea0
Sep 27 13:50:47 dist-test-slave-q25r krb5kdc[31319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517047, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:50:47.414453 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:47.396038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:46301 (local address 127.20.133.126:45029)
0927 13:50:47.396642 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:47.396648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:47.397934 (+ 1286us) server_negotiation.cc:407] Connection header received
0927 13:50:47.399912 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:47.399918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:47.400019 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:47.400120 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:47.401789 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:47.402633 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:47.404373 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:47.404861 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:47.404978 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:47.408554 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:47.408561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:47.408564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:47.408601 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:47.411280 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:47.412050 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:47.412055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:47.412057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:47.412186 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:47.412672 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:47.412675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:47.412677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:47.412967 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:47.413106 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:47.413840 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 13:50:47.414078 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":317,"threads_started":1}
I20260927 13:50:47.415441 31690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:47.396850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45029 (local address 127.20.133.66:46301)
0927 13:50:47.397737 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:47.397782 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:47.399606 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:47.400425 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:47.400434 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:47.401022 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:47.401617 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:47.401634 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:47.402789 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:47.402793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:47.404213 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:47.404225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:47.405105 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:47.405113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:47.405361 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:47.406094 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:47.406116 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:47.408280 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:47.411439 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:47.411449 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:47.411467 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:47.411920 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:47.412292 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:47.412297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:47.412301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:47.412553 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:47.413109 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:47.413118 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:47.413476 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 13:50:47.413726 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 13:50:47.414283 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":342,"threads_started":1}
I20260927 13:50:47.417379 31687 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45029
I20260927 13:50:47.417855 31687 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:47.418960 31687 heartbeater.cc:511] Master 127.20.133.126:45029 requested a full tablet report, sending...
I20260927 13:50:47.420987 31364 ts_manager.cc:194] Registered new tserver with Master: bdd8aaa37e374822b81d311ccc2110c8 (127.20.133.66:46747)
I20260927 13:50:47.421875 31364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:46301
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:50:47.464007 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45029
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:50:47.798720 31693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:50:47.799006 31693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:50:47.799115 31693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:50:47.806758 31693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:50:47.806887 31693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:50:47.807041 31693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:50:47.817157 31693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:50:47.818473 31693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:50:47.819736 31693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:50:47.833685 31698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:47.833585 31701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:50:47.833628 31699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:50:47.834316 31693 server_base.cc:1034] running on GCE node
I20260927 13:50:47.835168 31693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:50:47.836248 31693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:50:47.837456 31693 hybrid_clock.cc:648] HybridClock initialized: now 1790517047837400 us; error 72 us; skew 500 ppm
I20260927 13:50:47.837783 31693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:50:47 dist-test-slave-q25r krb5kdc[31319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517047, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:50:47.841130 31693 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:50:47.842753 31693 webserver.cc:460] Webserver started at http://127.20.133.67:39253/ using document root <none> and password file <none>
I20260927 13:50:47.843518 31693 fs_manager.cc:359] Metadata directory not provided
I20260927 13:50:47.843649 31693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:50:47.843946 31693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:50:47.847040 31693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "84d45e64892c4d99a306efaefb2f8ba1"
format_stamp: "Formatted at 2026-09-27 13:50:47 on dist-test-slave-q25r"
server_key: "e4214268295ca7b495d1c4caff67f8f8"
server_key_iv: "ade2515c11469956bbfd3c54b212390e"
server_key_version: "encryptionkey@0"
I20260927 13:50:47.847806 31693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "84d45e64892c4d99a306efaefb2f8ba1"
format_stamp: "Formatted at 2026-09-27 13:50:47 on dist-test-slave-q25r"
server_key: "e4214268295ca7b495d1c4caff67f8f8"
server_key_iv: "ade2515c11469956bbfd3c54b212390e"
server_key_version: "encryptionkey@0"
I20260927 13:50:47.851809 31553 heartbeater.cc:503] Master 127.20.133.126:45029 was elected leader, sending a full tablet report...
I20260927 13:50:47.854058 31693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 13:50:47.858127 31708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:47.859735 31693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:50:47.859906 31693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "84d45e64892c4d99a306efaefb2f8ba1"
format_stamp: "Formatted at 2026-09-27 13:50:47 on dist-test-slave-q25r"
server_key: "e4214268295ca7b495d1c4caff67f8f8"
server_key_iv: "ade2515c11469956bbfd3c54b212390e"
server_key_version: "encryptionkey@0"
I20260927 13:50:47.860082 31693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:50:47.912446 31693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:50:47.917352 31693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:50:47.917622 31693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:50:47.918586 31693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:50:47.919972 31693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:50:47.920050 31693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:47.920147 31693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:50:47.920193 31693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:50:47.955120 31693 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:36843
I20260927 13:50:47.955238 31820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:36843 every 8 connection(s)
I20260927 13:50:47.956558 31693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:50:47.964282 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 31693
I20260927 13:50:47.964612 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:50:47.965140 21013 external_mini_cluster.cc:1428] Setting key ce0b684203768d9ebffbeee0d54dd2d2
Sep 27 13:50:47 dist-test-slave-q25r krb5kdc[31319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517047, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:50:47.980638 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:47.959827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:34111 (local address 127.20.133.126:45029)
0927 13:50:47.960652 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:47.960658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:47.961711 (+ 1053us) server_negotiation.cc:407] Connection header received
0927 13:50:47.963795 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:47.963801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:47.963924 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:47.964045 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:47.966058 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:47.967222 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:47.969193 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:47.969760 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:47.969886 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:47.973940 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:47.973947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:47.973951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:47.974002 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:47.976910 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:47.977907 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:47.977911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:47.977914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:47.978047 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:47.978582 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:47.978586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:47.978588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:47.978970 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:47.979111 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:47.979972 (+ 861us) server_negotiation.cc:299] Negotiation successful
0927 13:50:47.980193 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":336,"threads_started":1}
I20260927 13:50:47.981643 31824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:47.960551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45029 (local address 127.20.133.67:34111)
0927 13:50:47.961503 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:47.961560 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:47.963479 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:47.964317 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:47.964326 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:47.965078 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:50:47.965762 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:47.965789 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:47.967461 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:47.967467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:47.968966 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:47.968984 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:47.969996 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:47.970001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:47.970238 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:47.971058 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:50:47.971086 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:50:47.973622 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:50:47.977135 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:47.977144 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:47.977161 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:47.977748 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:47.978186 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:50:47.978190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:50:47.978193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:50:47.978419 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:50:47.979126 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:50:47.979133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:50:47.979538 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 13:50:47.979793 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 13:50:47.980384 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":346,"threads_started":1}
I20260927 13:50:47.983744 31821 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45029
I20260927 13:50:47.984119 31821 heartbeater.cc:464] Registering TS with master...
I20260927 13:50:47.984972 31821 heartbeater.cc:511] Master 127.20.133.126:45029 requested a full tablet report, sending...
I20260927 13:50:47.986827 31364 ts_manager.cc:194] Registered new tserver with Master: 84d45e64892c4d99a306efaefb2f8ba1 (127.20.133.67:36843)
I20260927 13:50:47.987912 31364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:34111
I20260927 13:50:47.995632 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:50:48.014792 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.001345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.20.133.126:45029)
0927 13:50:48.001643 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:48.001651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:48.002159 (+ 508us) server_negotiation.cc:407] Connection header received
0927 13:50:48.002360 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:48.002366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:48.002453 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:48.002540 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:48.004099 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.004925 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.006162 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.006593 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.006705 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.009235 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:48.009240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:48.009244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:48.009282 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:48.012036 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:48.012778 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:48.012784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:48.012787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:48.012907 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:48.013284 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:48.013289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:48.013292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:48.013643 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:48.013912 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:48.014234 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 13:50:48.014469 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 13:50:48.020058 31364 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:50:48.028288 31364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:50:48.064734 31834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.044904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:46747 (local address 127.0.0.1:58974)
0927 13:50:48.053820 (+ 8916us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:48.053838 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:48.053968 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:48.056002 (+ 2034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:48.056007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:48.056031 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:48.056508 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:48.056520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.058483 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.058489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:48.061048 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:48.061064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.063346 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.063354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:48.063611 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.063668 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 13:50:48.063801 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 13:50:48.064139 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8796,"mutex_wait_us":2,"thread_start_us":7617,"threads_started":1}
I20260927 13:50:48.065231 31836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.045131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.20.133.65:43029)
0927 13:50:48.054109 (+ 8978us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:48.054118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:48.054147 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:50:48.054242 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:48.054249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:48.054509 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:48.054714 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:48.055583 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.057399 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.062201 (+ 4802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.063279 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.063418 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.063667 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:48.064728 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0927 13:50:48.064923 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8769,"thread_start_us":8560,"threads_started":1}
I20260927 13:50:48.064793 31835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.044757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:43029 (local address 127.0.0.1:49648)
0927 13:50:48.053018 (+ 8261us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:48.053085 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:48.053310 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:48.054752 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:48.054760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:48.054791 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:48.055277 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:48.055291 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.057672 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.057681 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:48.061974 (+ 4293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:48.061985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.063814 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.063821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:48.064001 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.064048 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 13:50:48.064142 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 13:50:48.064290 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7951,"mutex_wait_us":1572,"thread_start_us":8775,"threads_started":1}
I20260927 13:50:48.066546 31837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.045835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.20.133.66:46747)
0927 13:50:48.055483 (+ 9648us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:48.055493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:48.055517 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:50:48.055610 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:48.055615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:48.055815 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:48.055997 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:48.056692 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.058298 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.061240 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.062989 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.065476 (+ 2487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.065849 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:48.065951 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 13:50:48.066178 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9449,"thread_start_us":9269,"threads_started":1}
I20260927 13:50:48.070196 31838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.048924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:36843 (local address 127.0.0.1:35080)
0927 13:50:48.060127 (+ 11203us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:48.060147 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:48.060292 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:48.061433 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:48.061438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:48.061467 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:48.061906 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:48.061919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.063916 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.063921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:48.066193 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:48.066212 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.069333 (+ 3121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.069340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:48.069541 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.069578 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 13:50:48.069688 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 13:50:48.069838 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11049,"thread_start_us":10886,"threads_started":1}
I20260927 13:50:48.070240 31839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.049665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.20.133.67:36843)
0927 13:50:48.059619 (+ 9954us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:48.059629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:48.060429 (+ 800us) server_negotiation.cc:407] Connection header received
0927 13:50:48.060619 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:48.060627 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:48.060922 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:48.061144 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:48.062217 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.063752 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.066439 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.068890 (+ 2451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.069026 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.069260 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:48.069755 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 13:50:48.069923 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9751,"thread_start_us":9577,"threads_started":1}
I20260927 13:50:48.069682 31488 tablet_service.cc:1467] Processing CreateTablet for tablet 41e8623d32ab4b23b33d61ba55aef96c (DEFAULT_TABLE table=test-table [id=73712734be5e46a9b70080fdc809b397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:48.070173 31622 tablet_service.cc:1467] Processing CreateTablet for tablet 41e8623d32ab4b23b33d61ba55aef96c (DEFAULT_TABLE table=test-table [id=73712734be5e46a9b70080fdc809b397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:48.071293 31488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e8623d32ab4b23b33d61ba55aef96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:48.071406 31622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e8623d32ab4b23b33d61ba55aef96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:48.074406 31756 tablet_service.cc:1467] Processing CreateTablet for tablet 41e8623d32ab4b23b33d61ba55aef96c (DEFAULT_TABLE table=test-table [id=73712734be5e46a9b70080fdc809b397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:50:48.075845 31756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e8623d32ab4b23b33d61ba55aef96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:50:48.083223 31840 tablet_bootstrap.cc:492] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: Bootstrap starting.
I20260927 13:50:48.085742 31841 tablet_bootstrap.cc:492] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: Bootstrap starting.
I20260927 13:50:48.088644 31840 tablet_bootstrap.cc:654] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:48.089771 31841 tablet_bootstrap.cc:654] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:48.090088 31840 log.cc:824] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:48.090572 31842 tablet_bootstrap.cc:492] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: Bootstrap starting.
I20260927 13:50:48.091126 31841 log.cc:824] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:48.094116 31842 tablet_bootstrap.cc:654] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: Neither blocks nor log segments found. Creating new log.
I20260927 13:50:48.094822 31841 tablet_bootstrap.cc:492] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: No bootstrap required, opened a new log
I20260927 13:50:48.095140 31841 ts_tablet_manager.cc:1397] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 13:50:48.095319 31842 log.cc:824] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: Log is configured to *not* fsync() on all Append() calls
I20260927 13:50:48.098640 31840 tablet_bootstrap.cc:492] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: No bootstrap required, opened a new log
I20260927 13:50:48.098640 31842 tablet_bootstrap.cc:492] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: No bootstrap required, opened a new log
I20260927 13:50:48.098932 31842 ts_tablet_manager.cc:1397] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 13:50:48.098932 31840 ts_tablet_manager.cc:1397] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260927 13:50:48.103099 31841 raft_consensus.cc:348] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928186b6254443297374842b1b0374d" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43029 } } peers { permanent_uuid: "bdd8aaa37e374822b81d311ccc2110c8" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46747 } } peers { permanent_uuid: "84d45e64892c4d99a306efaefb2f8ba1" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36843 } }
I20260927 13:50:48.103514 31841 raft_consensus.cc:374] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:48.103631 31841 raft_consensus.cc:729] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd8aaa37e374822b81d311ccc2110c8, State: Initialized, Role: FOLLOWER
I20260927 13:50:48.104235 31841 consensus_queue.cc:260] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928186b6254443297374842b1b0374d" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43029 } } peers { permanent_uuid: "bdd8aaa37e374822b81d311ccc2110c8" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46747 } } peers { permanent_uuid: "84d45e64892c4d99a306efaefb2f8ba1" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36843 } }
I20260927 13:50:48.105399 31687 heartbeater.cc:503] Master 127.20.133.126:45029 was elected leader, sending a full tablet report...
I20260927 13:50:48.106129 31841 ts_tablet_manager.cc:1428] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 13:50:48.109781 31840 raft_consensus.cc:348] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928186b6254443297374842b1b0374d" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43029 } } peers { permanent_uuid: "bdd8aaa37e374822b81d311ccc2110c8" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46747 } } peers { permanent_uuid: "84d45e64892c4d99a306efaefb2f8ba1" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36843 } }
I20260927 13:50:48.109781 31842 raft_consensus.cc:348] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928186b6254443297374842b1b0374d" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43029 } } peers { permanent_uuid: "bdd8aaa37e374822b81d311ccc2110c8" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46747 } } peers { permanent_uuid: "84d45e64892c4d99a306efaefb2f8ba1" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36843 } }
I20260927 13:50:48.110270 31840 raft_consensus.cc:374] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:48.110273 31842 raft_consensus.cc:374] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:50:48.110381 31842 raft_consensus.cc:729] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d45e64892c4d99a306efaefb2f8ba1, State: Initialized, Role: FOLLOWER
I20260927 13:50:48.110375 31840 raft_consensus.cc:729] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0928186b6254443297374842b1b0374d, State: Initialized, Role: FOLLOWER
I20260927 13:50:48.111037 31840 consensus_queue.cc:260] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928186b6254443297374842b1b0374d" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43029 } } peers { permanent_uuid: "bdd8aaa37e374822b81d311ccc2110c8" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46747 } } peers { permanent_uuid: "84d45e64892c4d99a306efaefb2f8ba1" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36843 } }
I20260927 13:50:48.111037 31842 consensus_queue.cc:260] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928186b6254443297374842b1b0374d" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43029 } } peers { permanent_uuid: "bdd8aaa37e374822b81d311ccc2110c8" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46747 } } peers { permanent_uuid: "84d45e64892c4d99a306efaefb2f8ba1" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36843 } }
I20260927 13:50:48.112303 31821 heartbeater.cc:503] Master 127.20.133.126:45029 was elected leader, sending a full tablet report...
I20260927 13:50:48.113296 31840 ts_tablet_manager.cc:1428] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260927 13:50:48.114157 31842 ts_tablet_manager.cc:1428] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260927 13:50:48.145282 31688 tablet.cc:2369] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:50:48.208256 31822 tablet.cc:2369] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:50:48.317265 31554 tablet.cc:2369] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:50:48.380280 31847 raft_consensus.cc:482] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:50:48.380617 31847 raft_consensus.cc:504] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928186b6254443297374842b1b0374d" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43029 } } peers { permanent_uuid: "bdd8aaa37e374822b81d311ccc2110c8" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46747 } } peers { permanent_uuid: "84d45e64892c4d99a306efaefb2f8ba1" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36843 } }
I20260927 13:50:48.382331 31847 leader_election.cc:288] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0928186b6254443297374842b1b0374d (127.20.133.65:43029), bdd8aaa37e374822b81d311ccc2110c8 (127.20.133.66:46747)
I20260927 13:50:48.388839 31836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.382871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:51737 (local address 127.20.133.65:43029)
0927 13:50:48.383206 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:48.383212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:48.383549 (+ 337us) server_negotiation.cc:407] Connection header received
0927 13:50:48.383648 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:48.383654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:48.383735 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:48.383844 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:48.384260 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.385094 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.386942 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.387769 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.387968 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.388068 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:48.388401 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 13:50:48.388540 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260927 13:50:48.388919 31824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.382793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:43029 (local address 127.20.133.67:51737)
0927 13:50:48.383227 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:48.383265 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:48.383420 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:48.383880 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:48.383885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:48.383914 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:48.384101 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:48.384110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.385246 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.385253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:48.386762 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:48.386776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.387946 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.387954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:48.388199 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.388227 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 13:50:48.388355 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 13:50:48.388522 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":58}
I20260927 13:50:48.388917 31837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.383126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:60177 (local address 127.20.133.66:46747)
0927 13:50:48.383453 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:48.383460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:48.383777 (+ 317us) server_negotiation.cc:407] Connection header received
0927 13:50:48.383921 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:48.383925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:48.383984 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:48.384068 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:50:48.384668 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.385417 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.387141 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.387990 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.388161 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.388236 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:48.388537 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 13:50:48.388669 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:50:48.389154 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.382917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:46747 (local address 127.20.133.67:60177)
0927 13:50:48.383672 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 13:50:48.383687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:50:48.383797 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:50:48.384105 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:50:48.384110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:50:48.384134 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:50:48.384502 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:48.384513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.385567 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.385571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:48.386985 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:50:48.386999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.388199 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.388205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:50:48.388392 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.388418 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 13:50:48.388504 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 13:50:48.388635 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":6,"thread_start_us":472,"threads_started":1}
I20260927 13:50:48.390193 31508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e8623d32ab4b23b33d61ba55aef96c" candidate_uuid: "84d45e64892c4d99a306efaefb2f8ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0928186b6254443297374842b1b0374d" is_pre_election: true
I20260927 13:50:48.390196 31642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e8623d32ab4b23b33d61ba55aef96c" candidate_uuid: "84d45e64892c4d99a306efaefb2f8ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd8aaa37e374822b81d311ccc2110c8" is_pre_election: true
I20260927 13:50:48.390798 31508 raft_consensus.cc:2454] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d45e64892c4d99a306efaefb2f8ba1 in term 0.
I20260927 13:50:48.390806 31642 raft_consensus.cc:2454] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d45e64892c4d99a306efaefb2f8ba1 in term 0.
I20260927 13:50:48.391593 31712 leader_election.cc:302] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d45e64892c4d99a306efaefb2f8ba1, bdd8aaa37e374822b81d311ccc2110c8; no voters:
I20260927 13:50:48.392045 31847 raft_consensus.cc:2781] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:50:48.392215 31847 raft_consensus.cc:482] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:50:48.392319 31847 raft_consensus.cc:3037] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:48.395493 31847 raft_consensus.cc:504] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928186b6254443297374842b1b0374d" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43029 } } peers { permanent_uuid: "bdd8aaa37e374822b81d311ccc2110c8" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46747 } } peers { permanent_uuid: "84d45e64892c4d99a306efaefb2f8ba1" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36843 } }
I20260927 13:50:48.396286 31847 leader_election.cc:288] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [CANDIDATE]: Term 1 election: Requested vote from peers 0928186b6254443297374842b1b0374d (127.20.133.65:43029), bdd8aaa37e374822b81d311ccc2110c8 (127.20.133.66:46747)
I20260927 13:50:48.397015 31508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e8623d32ab4b23b33d61ba55aef96c" candidate_uuid: "84d45e64892c4d99a306efaefb2f8ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0928186b6254443297374842b1b0374d"
I20260927 13:50:48.396999 31642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e8623d32ab4b23b33d61ba55aef96c" candidate_uuid: "84d45e64892c4d99a306efaefb2f8ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd8aaa37e374822b81d311ccc2110c8"
I20260927 13:50:48.397239 31642 raft_consensus.cc:3037] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:48.397231 31508 raft_consensus.cc:3037] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:50:48.400502 31642 raft_consensus.cc:2454] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d45e64892c4d99a306efaefb2f8ba1 in term 1.
I20260927 13:50:48.400491 31508 raft_consensus.cc:2454] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d45e64892c4d99a306efaefb2f8ba1 in term 1.
I20260927 13:50:48.401058 31712 leader_election.cc:302] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d45e64892c4d99a306efaefb2f8ba1, bdd8aaa37e374822b81d311ccc2110c8; no voters:
I20260927 13:50:48.401407 31847 raft_consensus.cc:2781] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:50:48.401878 31847 raft_consensus.cc:686] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 1 LEADER]: Becoming Leader. State: Replica: 84d45e64892c4d99a306efaefb2f8ba1, State: Running, Role: LEADER
I20260927 13:50:48.402441 31847 consensus_queue.cc:237] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928186b6254443297374842b1b0374d" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43029 } } peers { permanent_uuid: "bdd8aaa37e374822b81d311ccc2110c8" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46747 } } peers { permanent_uuid: "84d45e64892c4d99a306efaefb2f8ba1" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36843 } }
I20260927 13:50:48.407519 31364 catalog_manager.cc:5495] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d45e64892c4d99a306efaefb2f8ba1 (127.20.133.67). New cstate: current_term: 1 leader_uuid: "84d45e64892c4d99a306efaefb2f8ba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928186b6254443297374842b1b0374d" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdd8aaa37e374822b81d311ccc2110c8" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d45e64892c4d99a306efaefb2f8ba1" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36843 } health_report { overall_health: HEALTHY } } }
I20260927 13:50:48.438081 31839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.432302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.20.133.67:36843)
0927 13:50:48.432554 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:48.432561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:48.432585 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:50:48.432704 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:48.432709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:48.432814 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:48.432970 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:50:48.433446 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.434236 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.435848 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.436260 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.436420 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.436899 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:50:48.437486 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:50:48.437619 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:48.437738 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 13:50:48.437826 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 13:50:48.446440 31508 raft_consensus.cc:1261] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d [term 1 FOLLOWER]: Refusing update from remote peer 84d45e64892c4d99a306efaefb2f8ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:48.446445 31642 raft_consensus.cc:1261] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8 [term 1 FOLLOWER]: Refusing update from remote peer 84d45e64892c4d99a306efaefb2f8ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:50:48.447425 31850 consensus_queue.cc:1035] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdd8aaa37e374822b81d311ccc2110c8" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:48.447773 31847 consensus_queue.cc:1035] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0928186b6254443297374842b1b0374d" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:50:48.459335 31853 mvcc.cc:205] Tried to move back new op lower bound from 7333957830426591232 to 7333957830261207040. Current Snapshot: MvccSnapshot[applied={T|T < 7333957830426591232}]
I20260927 13:50:48.473605 31363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:48.473894 31363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:50:48.477756 31363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6ecdff4d19447f7a6fcb7e5f37d20cb: Sending DeleteTablet for 3 replicas of tablet 41e8623d32ab4b23b33d61ba55aef96c
I20260927 13:50:48.479174 31488 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e8623d32ab4b23b33d61ba55aef96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:48 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:49648
I20260927 13:50:48.479162 31622 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e8623d32ab4b23b33d61ba55aef96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:48 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:58974
I20260927 13:50:48.480293 31756 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e8623d32ab4b23b33d61ba55aef96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:50:48 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:35080
I20260927 13:50:48.480791 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 31425
I20260927 13:50:48.488860 31865 tablet_replica.cc:330] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: stopping tablet replica
I20260927 13:50:48.489055 31864 tablet_replica.cc:330] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: stopping tablet replica
I20260927 13:50:48.489423 31865 raft_consensus.cc:2229] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:50:48.489886 31863 tablet_replica.cc:330] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: stopping tablet replica
I20260927 13:50:48.490149 31865 raft_consensus.cc:2258] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:48.490334 31863 raft_consensus.cc:2229] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:48.490916 31863 raft_consensus.cc:2258] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:48.491612 31864 raft_consensus.cc:2229] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:50:48.492180 31864 raft_consensus.cc:2258] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:50:48.492940 31865 ts_tablet_manager.cc:1905] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:48.494341 31864 ts_tablet_manager.cc:1905] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:50:48.494470 31863 ts_tablet_manager.cc:1905] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 13:50:48 dist-test-slave-q25r krb5kdc[31319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:50:48.500789 31865 ts_tablet_manager.cc:1918] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:48.500983 31865 log.cc:1196] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/41e8623d32ab4b23b33d61ba55aef96c
I20260927 13:50:48.501864 31863 ts_tablet_manager.cc:1918] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:48.502053 31863 log.cc:1196] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/41e8623d32ab4b23b33d61ba55aef96c
I20260927 13:50:48.502609 31863 ts_tablet_manager.cc:1939] T 41e8623d32ab4b23b33d61ba55aef96c P bdd8aaa37e374822b81d311ccc2110c8: Deleting consensus metadata
I20260927 13:50:48.503271 31865 ts_tablet_manager.cc:1939] T 41e8623d32ab4b23b33d61ba55aef96c P 84d45e64892c4d99a306efaefb2f8ba1: Deleting consensus metadata
I20260927 13:50:48.505358 31349 catalog_manager.cc:4841] TS 84d45e64892c4d99a306efaefb2f8ba1 (127.20.133.67:36843): tablet 41e8623d32ab4b23b33d61ba55aef96c (table test-table [id=73712734be5e46a9b70080fdc809b397]) successfully deleted
I20260927 13:50:48.506093 31351 catalog_manager.cc:4841] TS bdd8aaa37e374822b81d311ccc2110c8 (127.20.133.66:46747): tablet 41e8623d32ab4b23b33d61ba55aef96c (table test-table [id=73712734be5e46a9b70080fdc809b397]) successfully deleted
I20260927 13:50:48.507908 31864 ts_tablet_manager.cc:1918] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:50:48.508123 31864 log.cc:1196] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/41e8623d32ab4b23b33d61ba55aef96c
I20260927 13:50:48.508769 31864 ts_tablet_manager.cc:1939] T 41e8623d32ab4b23b33d61ba55aef96c P 0928186b6254443297374842b1b0374d: Deleting consensus metadata
I20260927 13:50:48.509658 31836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.482725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.20.133.65:43029)
0927 13:50:48.482939 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:48.482946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:48.488679 (+ 5733us) server_negotiation.cc:407] Connection header received
0927 13:50:48.489193 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:48.489200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:48.489294 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:48.489382 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:48.491430 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.492682 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.494567 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.495208 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.495325 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.502095 (+ 6770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:48.502116 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:48.502146 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:48.502205 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:48.506205 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:48.507040 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:48.507050 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:48.507053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:48.507248 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:48.507660 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:48.507665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:48.507669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:48.508097 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:48.508469 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:48.508795 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 13:50:48.509055 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:50:48.510351 31548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:48.510972 31350 catalog_manager.cc:4841] TS 0928186b6254443297374842b1b0374d (127.20.133.65:43029): tablet 41e8623d32ab4b23b33d61ba55aef96c (table test-table [id=73712734be5e46a9b70080fdc809b397]) successfully deleted
I20260927 13:50:48.601161 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 31425
I20260927 13:50:48.623646 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 31559
Sep 27 13:50:48 dist-test-slave-q25r krb5kdc[31319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:50:48.640121 31837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.624440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.20.133.66:46747)
0927 13:50:48.624594 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:48.624598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:48.624616 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:50:48.624808 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:48.624812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:48.624875 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:48.624948 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:48.626507 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.627222 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.629065 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.629487 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.629601 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.634434 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:48.634451 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:48.634467 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:48.634513 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:48.637291 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:48.638038 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:48.638046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:48.638048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:48.638185 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:48.638545 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:48.638549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:48.638551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:48.638956 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:48.639253 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:48.639559 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 13:50:48.639780 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 13:50:48.640733 31682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:48.731007 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 31559
I20260927 13:50:48.750835 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 31693
Sep 27 13:50:48 dist-test-slave-q25r krb5kdc[31319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:50:48.768167 31839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:50:48.751612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.20.133.67:36843)
0927 13:50:48.751812 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 13:50:48.751818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:50:48.751901 (+ 83us) server_negotiation.cc:407] Connection header received
0927 13:50:48.752092 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:50:48.752098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:50:48.752166 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:50:48.752252 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:50:48.753555 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.754398 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.755939 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:50:48.756542 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:50:48.756676 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:50:48.761056 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:50:48.761069 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:50:48.761084 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:50:48.761129 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:50:48.764996 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:48.765844 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:48.765855 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:48.765859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:48.766070 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:50:48.766436 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:50:48.766442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:50:48.766446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:50:48.766974 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:50:48.767319 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 13:50:48.767514 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 13:50:48.767780 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 13:50:48.769045 31816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:48.890383 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 31693
I20260927 13:50:48.906736 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 31332
I20260927 13:50:48.907969 31394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:50:49.018620 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 31332
2026-09-27T13:50:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4802 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:50:49 dist-test-slave-q25r krb5kdc[31873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:50:49 dist-test-slave-q25r krb5kdc[31873](info): set up 2 sockets
Sep 27 13:50:49 dist-test-slave-q25r krb5kdc[31873](info): commencing operation
krb5kdc: starting...
W20260927 13:50:50.061513 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:50:50 dist-test-slave-q25r krb5kdc[31873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:50:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.133.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.133.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/rangeradmin_127.20.133.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/rangeradmin_127.20.133.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.133.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.133.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/rangerlookup_127.20.133.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/rangerlookup_127.20.133.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.133.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.133.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/HTTP_127.20.133.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/HTTP_127.20.133.84@KRBTEST.COM.keytab.
I20260927 13:50:50.455670 21013 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskM7FxcK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/postgres -l logfile start
2026-09-27 13:50:53.028 UTC [31903] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 13:50:53.028 UTC [31903] LOG: listening on IPv4 address "127.20.133.84", port 53781
2026-09-27 13:50:53.033 UTC [31903] LOG: listening on Unix socket "/tmp/.s.PGSQL.53781"
2026-09-27 13:50:53.056 UTC [31906] LOG: database system was shut down at 2026-09-27 13:50:51 UTC
2026-09-27 13:50:53.060 UTC [31903] LOG: database system is ready to accept connections
I20260927 13:50:53.964783 21013 mini_postgres.cc:96] Postgres bound to 53781
2026-09-27 13:50:53.982 UTC [31914] FATAL: database "slave" does not exist
127.20.133.84:53781 - accepting connections
I20260927 13:50:53.983407 21013 mini_ranger.cc:161] Starting Ranger...
I20260927 13:50:54.010800 21013 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 13:50:54.531239 21013 mini_ranger.cc:87] Created ranger Postgres database
I20260927 13:50:54.531369 21013 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-admin
2026-09-27 13:50:54,814 [I] DB FLAVOR :POSTGRES
2026-09-27 13:50:54,814 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:50:54,814 [I] Checking connection..
2026-09-27 13:50:54,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 13:50:55,079 [I] Checking connection passed.
2026-09-27 13:50:55,079 [I] --------- Verifying version history table ---------
2026-09-27 13:50:55,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:50:55,329 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:50:55,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:50:55,571 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:50:55,572 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 13:50:55,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 13:50:55.835 UTC [31978] WARNING: there is no transaction in progress
2026-09-27 13:50:55,842 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 13:50:55,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:50:56,072 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 13:50:56,072 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:50:56,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 13:50:56,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 13:50:56,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q25r.c.gcp-upstream.internal','N') ;"
2026-09-27 13:50:56,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:50:57,071 [I] Table x_portal_user does not exist in database ranger
2026-09-27 13:50:57,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:50:57,314 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 13:50:57,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:50:57,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:50:57,822 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 13:50:57,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 13:50:58.772 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.789 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.804 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.819 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.836 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.853 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.908 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.915 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.928 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.939 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.950 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.964 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.974 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.981 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.988 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:58.995 UTC [32086] WARNING: there is no transaction in progress
2026-09-27 13:50:59,644 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 13:50:59,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 13:50:59,897 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 13:50:59,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:51:00,149 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 13:51:00,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:51:00,412 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 13:51:00,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:51:00,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:51:00,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q25r.c.gcp-upstream.internal';"
2026-09-27 13:51:01,138 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 13:51:01,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 13:51:01,390 [I] DB_PATCHES have already been applied
I20260927 13:51:01.397841 21013 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-admin:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskM7FxcK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 13:51:01.397958 21013 mini_ranger.cc:193] Using host: 127.20.133.84
I20260927 13:51:01.416586 21013 mini_ranger.cc:227] Ranger admin URL: http://127.20.133.84:42839
Sep 27, 2026 1:51:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:51:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:51:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 1:51:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 1:51:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 1:51:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 1:51:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 13:51:02.463186 21013 mini_ranger.cc:160] Time spent starting Ranger: real 8.480s user 0.003s sys 0.054s
Sep 27, 2026 1:51:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 1:51:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 1:51:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 1:51:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 1:51:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 13:51:25.786273 21013 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.133.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.133.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/rangerkms_127.20.133.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/rangerkms_127.20.133.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.133.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.133.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.133.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/HTTP_127.20.133.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.133.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/HTTP_127.20.133.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 13:51:25 dist-test-slave-q25r krb5kdc[31873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517085, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 13:51:25.970036 21013 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 13:51:25.994524 21013 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 13:51:26.904628 21013 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 13:51:26.904803 21013 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-kms
I20260927 13:51:26.904928 21013 mini_ranger_kms.cc:227] Using postgres at 127.20.133.84:53781
2026-09-27 13:51:27,337 [I] DB FLAVOR :POSTGRES
2026-09-27 13:51:27,338 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:51:27,338 [I] Checking connection
2026-09-27 13:51:27,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 13:51:27,619 [I] connection success
2026-09-27 13:51:27,619 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 13:51:27,619 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 13:51:27,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 13:51:27,884 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 13:51:27,884 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:51:27,884 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 13:51:27,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:53781/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 13:51:28,160 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 13:51:28.650938 21013 mini_ranger_kms.cc:310] Created kms service
I20260927 13:51:28.847596 21013 mini_ranger_kms.cc:326] Created kudu user
I20260927 13:51:28.890139 21013 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 13:51:28 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517085, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:51:29.322775 21013 mini_ranger_kms.cc:384] Added ranger policy
I20260927 13:51:29.323107 21013 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-kms:/tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 13:51:29.323165 21013 mini_ranger_kms.cc:240] Using host: 127.20.133.84
I20260927 13:51:29.344381 21013 mini_ranger_kms.cc:276] Ranger KMS PID: 32267
I20260927 13:51:29.344581 21013 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.133.84:47969
Sep 27, 2026 1:51:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:51:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:51:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 1:51:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 1:51:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 1:51:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 13:51:31.450366 21013 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.480s user 0.163s sys 0.299s
I20260927 13:51:31.450635 21013 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 13:51:31.450863 21013 mini_ranger_kms.cc:401] 127.20.133.84:47969/kms/v1/keys
Sep 27 13:51:31 dist-test-slave-q25r krb5kdc[31873](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790517091, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.133.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 13:51:31 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517091, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.133.84@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
Sep 27 13:51:32 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517091, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.133.84@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
Sep 27 13:51:34 dist-test-slave-q25r krb5kdc[31873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:51:34.960242 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:40231
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:40231
--ranger_config_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:47969/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:51:35.316573 32317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:51:35.316880 32317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:51:35.317018 32317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:51:35.324393 32317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:51:35.324518 32317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:51:35.324563 32317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:51:35.324613 32317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:51:35.324658 32317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:51:35.334932 32317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.133.84:47969/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:40231
--ranger_config_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:40231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:51:35.336231 32317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:51:35.337674 32317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:51:35.350028 32322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:51:35.349942 32325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:51:35.349963 32323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:51:35.351013 32317 server_base.cc:1034] running on GCE node
I20260927 13:51:35.351796 32317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:51:35.353569 32317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:51:35.354828 32317 hybrid_clock.cc:648] HybridClock initialized: now 1790517095354776 us; error 38 us; skew 500 ppm
I20260927 13:51:35.355253 32317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:51:35 dist-test-slave-q25r krb5kdc[31873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517095, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:51:35.359220 32317 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:51:35.361589 32317 webserver.cc:460] Webserver started at http://127.20.133.126:46205/ using document root <none> and password file <none>
I20260927 13:51:35.362362 32317 fs_manager.cc:359] Metadata directory not provided
I20260927 13:51:35.362491 32317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:51:35.362824 32317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:51:35 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517095, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:51:35.416548 32317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "fb03f4b5730d463ca6d3ec57b50f5599"
format_stamp: "Formatted at 2026-09-27 13:51:35 on dist-test-slave-q25r"
server_key: "eac2eb9e2c736c1fd66cee5b84be11a4"
server_key_iv: "edf5ed95649b590023ea63fc39fc77a5"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:35.417610 32317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "fb03f4b5730d463ca6d3ec57b50f5599"
format_stamp: "Formatted at 2026-09-27 13:51:35 on dist-test-slave-q25r"
server_key: "eac2eb9e2c736c1fd66cee5b84be11a4"
server_key_iv: "edf5ed95649b590023ea63fc39fc77a5"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:35.425328 32317 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260927 13:51:35.447552 32332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:51:35.449270 32317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:51:35.449522 32317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "fb03f4b5730d463ca6d3ec57b50f5599"
format_stamp: "Formatted at 2026-09-27 13:51:35 on dist-test-slave-q25r"
server_key: "eac2eb9e2c736c1fd66cee5b84be11a4"
server_key_iv: "edf5ed95649b590023ea63fc39fc77a5"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:35.449761 32317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:51:35.488996 32317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:51:35.493271 32317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:51:35.493584 32317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:51:35.511818 32317 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:40231
I20260927 13:51:35.511847 32383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:40231 every 8 connection(s)
I20260927 13:51:35.513299 32317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:51:35.517289 32384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:51:35.521370 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 32317
I20260927 13:51:35.521589 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:51:35.522635 21013 external_mini_cluster.cc:1428] Setting key c0e8c1b406594635fc46c471ae943b8e
I20260927 13:51:35.530189 32384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599: Bootstrap starting.
I20260927 13:51:35.533715 32384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599: Neither blocks nor log segments found. Creating new log.
I20260927 13:51:35.534873 32384 log.cc:824] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599: Log is configured to *not* fsync() on all Append() calls
I20260927 13:51:35.538791 32384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599: No bootstrap required, opened a new log
Sep 27 13:51:35 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:51:35.549790 32384 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb03f4b5730d463ca6d3ec57b50f5599" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40231 } }
I20260927 13:51:35.550258 32384 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:51:35.550371 32384 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb03f4b5730d463ca6d3ec57b50f5599, State: Initialized, Role: FOLLOWER
I20260927 13:51:35.550899 32387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:35.525278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.20.133.126:40231)
0927 13:51:35.526178 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:35.526192 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:35.526235 (+ 43us) server_negotiation.cc:407] Connection header received
0927 13:51:35.528680 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:35.528712 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:35.529197 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:35.529624 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:51:35.531114 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:35.532751 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:35.534206 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:35.535011 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:35.535216 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:35.541107 (+ 5891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:51:35.541124 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:51:35.541154 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:51:35.541225 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:51:35.545843 (+ 4618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:35.546754 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:35.546765 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:35.546775 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:35.546978 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:35.547440 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:35.547446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:35.547449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:35.548291 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:51:35.548653 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:35.549089 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 13:51:35.549534 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":405,"threads_started":1}
I20260927 13:51:35.551072 32384 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb03f4b5730d463ca6d3ec57b50f5599" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40231 } }
I20260927 13:51:35.551380 32384 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:51:35.551517 32384 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:51:35.551689 32384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:51:35.554983 32384 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb03f4b5730d463ca6d3ec57b50f5599" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40231 } }
I20260927 13:51:35.555553 32384 leader_election.cc:302] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb03f4b5730d463ca6d3ec57b50f5599; no voters:
I20260927 13:51:35.556260 32384 leader_election.cc:288] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:51:35.556439 32389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:51:35.558676 32384 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:51:35.560096 32389 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [term 1 LEADER]: Becoming Leader. State: Replica: fb03f4b5730d463ca6d3ec57b50f5599, State: Running, Role: LEADER
I20260927 13:51:35.560920 32389 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb03f4b5730d463ca6d3ec57b50f5599" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40231 } }
I20260927 13:51:35.565579 32390 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb03f4b5730d463ca6d3ec57b50f5599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb03f4b5730d463ca6d3ec57b50f5599" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40231 } } }
I20260927 13:51:35.566053 32390 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [sys.catalog]: This master's current role is: LEADER
I20260927 13:51:35.565640 32391 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb03f4b5730d463ca6d3ec57b50f5599. Latest consensus state: current_term: 1 leader_uuid: "fb03f4b5730d463ca6d3ec57b50f5599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb03f4b5730d463ca6d3ec57b50f5599" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40231 } } }
I20260927 13:51:35.566442 32391 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599 [sys.catalog]: This master's current role is: LEADER
I20260927 13:51:35.567835 32401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:51:35.569393 32384 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 13:51:35.584751 32401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:51:35.599385 32401 catalog_manager.cc:1347] Generated new cluster ID: 5c5df3332abc46e68aca896d2d6f4370
I20260927 13:51:35.599581 32401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:51:35.612588 32401 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:51:35.614712 32401 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:51:35.623759 32401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599: Generated new TSK 0
I20260927 13:51:35.625064 32401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:51:36.440191 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:47969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40231
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:51:36 dist-test-slave-q25r krb5kdc[31873](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790517096, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 13:51:36.938191 32429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:51:36.938557 32429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:51:36.938681 32429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:51:36.947048 32429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:51:36.947304 32429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:51:36.947525 32429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:51:36.964279 32429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.133.84:47969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:51:36.966439 32429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:51:36.968596 32429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:51:36.992655 32440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:51:36.993244 32439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:51:36.996322 32429 server_base.cc:1034] running on GCE node
W20260927 13:51:36.996842 32443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:51:36.997848 32429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:51:37.000672 32429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:51:37.004179 32429 hybrid_clock.cc:648] HybridClock initialized: now 1790517097003627 us; error 535 us; skew 500 ppm
I20260927 13:51:37.004751 32429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:51:37 dist-test-slave-q25r krb5kdc[31873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517097, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:51:37.010107 32429 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:51:37.012917 32429 webserver.cc:460] Webserver started at http://127.20.133.65:46681/ using document root <none> and password file <none>
I20260927 13:51:37.014219 32429 fs_manager.cc:359] Metadata directory not provided
I20260927 13:51:37.014390 32429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:51:37.014824 32429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:51:37 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517097, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:51:37.038741 32429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "476ce715c3714519886633752f9ccb63"
format_stamp: "Formatted at 2026-09-27 13:51:37 on dist-test-slave-q25r"
server_key: "75d18a23eb76aeec7a4249ed707fe5df"
server_key_iv: "efbc6fa7ce2246a2021a690ff661b163"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:37.042470 32429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "476ce715c3714519886633752f9ccb63"
format_stamp: "Formatted at 2026-09-27 13:51:37 on dist-test-slave-q25r"
server_key: "75d18a23eb76aeec7a4249ed707fe5df"
server_key_iv: "efbc6fa7ce2246a2021a690ff661b163"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:37.066462 32429 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.012s sys 0.004s
I20260927 13:51:37.098006 32449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:51:37.100185 32429 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.009s sys 0.000s
I20260927 13:51:37.100471 32429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "476ce715c3714519886633752f9ccb63"
format_stamp: "Formatted at 2026-09-27 13:51:37 on dist-test-slave-q25r"
server_key: "75d18a23eb76aeec7a4249ed707fe5df"
server_key_iv: "efbc6fa7ce2246a2021a690ff661b163"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:37.100780 32429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:51:37.140861 32429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 27 13:51:37 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517096, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:51:37.145656 32429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:51:37.145990 32429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:51:37.153764 32429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:51:37.160943 32429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:51:37.161065 32429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:51:37.161206 32429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:51:37.161268 32429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:51:37.312999 32429 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:45801
I20260927 13:51:37.313027 32561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:45801 every 8 connection(s)
I20260927 13:51:37.314751 32429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:51:37.324122 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 32429
I20260927 13:51:37.324365 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:51:37.325220 21013 external_mini_cluster.cc:1428] Setting key 5ffba009c15c84c6506863c75a55cff5
Sep 27 13:51:37 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517097, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
I20260927 13:51:37.370615 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:37.317759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:41475 (local address 127.20.133.126:40231)
0927 13:51:37.337184 (+ 19425us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:37.337191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:37.345219 (+ 8028us) server_negotiation.cc:407] Connection header received
0927 13:51:37.348135 (+ 2916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:37.348141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:37.348258 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:37.348391 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:51:37.351727 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:37.352898 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:37.356033 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:37.356726 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:37.356857 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:37.362311 (+ 5454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:51:37.362319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:51:37.362324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:51:37.362373 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:51:37.366134 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:37.367313 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:37.367319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:37.367322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:37.367482 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:37.368101 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:37.368106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:37.368109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:37.368526 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:51:37.368667 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:37.369904 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0927 13:51:37.370183 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19275,"thread_start_us":19427,"threads_started":1}
I20260927 13:51:37.373425 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:37.318413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:40231 (local address 127.20.133.65:41475)
0927 13:51:37.344993 (+ 26580us) negotiation.cc:107] Waiting for socket to connect
0927 13:51:37.345054 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 13:51:37.347679 (+ 2625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:51:37.349966 (+ 2287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:51:37.349979 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:51:37.350637 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:51:37.351477 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:37.351505 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:37.353134 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:37.353151 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:37.355829 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:37.355849 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:37.357304 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:37.357312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:37.357624 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:37.358579 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:51:37.358612 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:51:37.361936 (+ 3324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:51:37.366399 (+ 4463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:51:37.366410 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:51:37.366432 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:51:37.367114 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:51:37.367642 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:51:37.367647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:51:37.367651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:51:37.367977 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:51:37.368692 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:51:37.368701 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:51:37.369397 (+ 696us) client_negotiation.cc:769] Sending connection context
0927 13:51:37.370874 (+ 1477us) client_negotiation.cc:240] Negotiation successful
0927 13:51:37.371529 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26190,"thread_start_us":29658,"threads_started":1}
I20260927 13:51:37.375552 32562 heartbeater.cc:347] Connected to a master server at 127.20.133.126:40231
I20260927 13:51:37.375916 32562 heartbeater.cc:464] Registering TS with master...
I20260927 13:51:37.376861 32562 heartbeater.cc:511] Master 127.20.133.126:40231 requested a full tablet report, sending...
I20260927 13:51:37.379760 32349 ts_manager.cc:194] Registered new tserver with Master: 476ce715c3714519886633752f9ccb63 (127.20.133.65:45801)
I20260927 13:51:37.381852 32349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:41475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:51:37.411765 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:47969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40231
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:51:37 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517096, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
W20260927 13:51:37.925808 32568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:51:37.926131 32568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:51:37.926301 32568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:51:37.934084 32568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:51:37.934231 32568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:51:37.934430 32568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:51:37.944660 32568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.133.84:47969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:51:37.946018 32568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:51:37.947316 32568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:51:37.962725 32582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:51:37.963649 32583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:51:37.978055 32585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:51:37.978448 32568 server_base.cc:1034] running on GCE node
I20260927 13:51:37.979113 32568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:51:37.980258 32568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:51:37.981582 32568 hybrid_clock.cc:648] HybridClock initialized: now 1790517097981515 us; error 58 us; skew 500 ppm
I20260927 13:51:37.981940 32568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:51:37 dist-test-slave-q25r krb5kdc[31873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517097, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:51:37.985170 32568 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:51:37.987021 32568 webserver.cc:460] Webserver started at http://127.20.133.66:40325/ using document root <none> and password file <none>
I20260927 13:51:37.987756 32568 fs_manager.cc:359] Metadata directory not provided
I20260927 13:51:37.987875 32568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:51:37.988196 32568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:51:37 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517097, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:51:38.008147 32568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "c70b5a0610b94f70b6c55281f3346e37"
format_stamp: "Formatted at 2026-09-27 13:51:38 on dist-test-slave-q25r"
server_key: "90d36abbf5efdfcffaee22c627746db8"
server_key_iv: "0d8ad5ed3d6f1eacfcc4a9f0494c40b5"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:38.009189 32568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "c70b5a0610b94f70b6c55281f3346e37"
format_stamp: "Formatted at 2026-09-27 13:51:38 on dist-test-slave-q25r"
server_key: "90d36abbf5efdfcffaee22c627746db8"
server_key_iv: "0d8ad5ed3d6f1eacfcc4a9f0494c40b5"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:38.014803 32568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 13:51:38.034353 32592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:51:38.036813 32568 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260927 13:51:38.037048 32568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "c70b5a0610b94f70b6c55281f3346e37"
format_stamp: "Formatted at 2026-09-27 13:51:38 on dist-test-slave-q25r"
server_key: "90d36abbf5efdfcffaee22c627746db8"
server_key_iv: "0d8ad5ed3d6f1eacfcc4a9f0494c40b5"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:38.037333 32568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:51:38.066218 32568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:51:38.071698 32568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:51:38.072045 32568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:51:38.073315 32568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:51:38.075145 32568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:51:38.075249 32568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:51:38.075412 32568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:51:38.075469 32568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:51:38.120798 32568 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:41681
I20260927 13:51:38.120837 32704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:41681 every 8 connection(s)
I20260927 13:51:38.122718 32568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:51:38.125399 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 32568
I20260927 13:51:38.125586 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:51:38.126101 21013 external_mini_cluster.cc:1428] Setting key baf94091dfc5f5e5d0c408ec0d5e4792
Sep 27 13:51:38 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517097, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
I20260927 13:51:38.178998 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:38.139053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:40231 (local address 127.20.133.66:38501)
0927 13:51:38.152916 (+ 13863us) negotiation.cc:107] Waiting for socket to connect
0927 13:51:38.152977 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 13:51:38.155633 (+ 2656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:51:38.157385 (+ 1752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:51:38.157400 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:51:38.158084 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:51:38.158926 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:38.158955 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.160598 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.160605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:38.162740 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:38.162762 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.164172 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.164180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:38.164587 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:38.165450 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:51:38.165483 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:51:38.168466 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:51:38.173211 (+ 4745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:51:38.173222 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:51:38.173244 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:51:38.173923 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:51:38.174492 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:51:38.174497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:51:38.174501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:51:38.174794 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:51:38.175792 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:51:38.175801 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:51:38.176304 (+ 503us) client_negotiation.cc:769] Sending connection context
0927 13:51:38.176681 (+ 377us) client_negotiation.cc:240] Negotiation successful
0927 13:51:38.177318 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13498,"thread_start_us":16914,"threads_started":1}
I20260927 13:51:38.180866 32707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:38.138179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:38501 (local address 127.20.133.126:40231)
0927 13:51:38.152204 (+ 14025us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:38.152212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:38.153587 (+ 1375us) server_negotiation.cc:407] Connection header received
0927 13:51:38.156866 (+ 3279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:38.156872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:38.156965 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:38.157097 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:51:38.159160 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.160291 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.162950 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.163674 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.163787 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:38.168809 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:51:38.168818 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:51:38.168823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:51:38.168869 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:51:38.172915 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:38.174131 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:38.174138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:38.174141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:38.174326 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:38.174928 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:38.174934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:38.174937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:38.175350 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:51:38.175514 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:38.180069 (+ 4555us) server_negotiation.cc:299] Negotiation successful
0927 13:51:38.180454 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13848,"thread_start_us":14154,"threads_started":1}
I20260927 13:51:38.182614 32705 heartbeater.cc:347] Connected to a master server at 127.20.133.126:40231
I20260927 13:51:38.182945 32705 heartbeater.cc:464] Registering TS with master...
I20260927 13:51:38.183830 32705 heartbeater.cc:511] Master 127.20.133.126:40231 requested a full tablet report, sending...
I20260927 13:51:38.185674 32349 ts_manager.cc:194] Registered new tserver with Master: c70b5a0610b94f70b6c55281f3346e37 (127.20.133.66:41681)
I20260927 13:51:38.186782 32349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:38501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:51:38.212791 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:47969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40231
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:43975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:51:38.386273 32562 heartbeater.cc:503] Master 127.20.133.126:40231 was elected leader, sending a full tablet report...
W20260927 13:51:38.664943 32711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:51:38.665225 32711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:51:38.665339 32711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:51:38.672784 32711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:51:38.672911 32711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:51:38.673056 32711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:51:38.683732 32711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:43975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.133.84:47969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:51:38.685062 32711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:51:38.686404 32711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:51:38.701383 32717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:51:38.701558 32716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:51:38.701793 32719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:51:38.702082 32711 server_base.cc:1034] running on GCE node
I20260927 13:51:38.702790 32711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:51:38.703943 32711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:51:38.705183 32711 hybrid_clock.cc:648] HybridClock initialized: now 1790517098705116 us; error 54 us; skew 500 ppm
I20260927 13:51:38.705550 32711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:51:38 dist-test-slave-q25r krb5kdc[31873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517098, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:51:38.709201 32711 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:51:38.710723 32711 webserver.cc:460] Webserver started at http://127.20.133.67:42339/ using document root <none> and password file <none>
I20260927 13:51:38.711395 32711 fs_manager.cc:359] Metadata directory not provided
I20260927 13:51:38.711499 32711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:51:38.711795 32711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:51:38 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517098, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:51:38.730006 32711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "f28051604766451fa742ab8c16c3b164"
format_stamp: "Formatted at 2026-09-27 13:51:38 on dist-test-slave-q25r"
server_key: "1a01696e96dc7e858fb146f3574cabc4"
server_key_iv: "f0028b1989052a05f99e4ff11ee16e71"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:38.731235 32711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "f28051604766451fa742ab8c16c3b164"
format_stamp: "Formatted at 2026-09-27 13:51:38 on dist-test-slave-q25r"
server_key: "1a01696e96dc7e858fb146f3574cabc4"
server_key_iv: "f0028b1989052a05f99e4ff11ee16e71"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:38.739609 32711 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s
I20260927 13:51:38.756588 32726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:51:38.758210 32711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:51:38.758366 32711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "f28051604766451fa742ab8c16c3b164"
format_stamp: "Formatted at 2026-09-27 13:51:38 on dist-test-slave-q25r"
server_key: "1a01696e96dc7e858fb146f3574cabc4"
server_key_iv: "f0028b1989052a05f99e4ff11ee16e71"
server_key_version: "kuduclusterkey@0"
I20260927 13:51:38.758605 32711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:51:38.792086 32711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:51:38.796409 32711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:51:38.796732 32711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:51:38.797816 32711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:51:38.799310 32711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:51:38.799391 32711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:51:38.799515 32711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:51:38.799569 32711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:51:38.833071 32711 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:46615
I20260927 13:51:38.833114 370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:46615 every 8 connection(s)
I20260927 13:51:38.834439 32711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:51:38.841495 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 32711
I20260927 13:51:38.841662 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:51:38.842093 21013 external_mini_cluster.cc:1428] Setting key 302b4344bcf654afa59b6cd97d6681ee
Sep 27 13:51:38 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517098, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:51:38.855598 373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:38.836917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:50887 (local address 127.20.133.126:40231)
0927 13:51:38.837366 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:38.837370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:38.838262 (+ 892us) server_negotiation.cc:407] Connection header received
0927 13:51:38.839965 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:38.839970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:38.840051 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:38.840155 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:51:38.842556 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.843759 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.845699 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.846154 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.846276 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:38.849669 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:51:38.849674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:51:38.849677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:51:38.849715 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:51:38.852416 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:38.853193 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:38.853197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:38.853199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:38.853342 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:38.853787 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:38.853792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:38.853796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:38.854083 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:51:38.854226 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:38.854987 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 13:51:38.855236 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260927 13:51:38.856642 374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:38.837368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:40231 (local address 127.20.133.67:50887)
0927 13:51:38.838091 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 13:51:38.838133 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:51:38.839715 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:51:38.840573 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:51:38.840588 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:51:38.841254 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:51:38.842117 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:38.842144 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.843953 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.843958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:38.845550 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:38.845569 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.846393 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.846400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:38.846648 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:38.847335 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:51:38.847358 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:51:38.849432 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:51:38.852584 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:51:38.852592 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:51:38.852608 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:51:38.853064 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:51:38.853437 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:51:38.853442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:51:38.853446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:51:38.853670 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:51:38.854207 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:51:38.854216 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:51:38.854634 (+ 418us) client_negotiation.cc:769] Sending connection context
0927 13:51:38.854875 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 13:51:38.855408 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":299,"threads_started":1}
I20260927 13:51:38.858063 371 heartbeater.cc:347] Connected to a master server at 127.20.133.126:40231
I20260927 13:51:38.858352 371 heartbeater.cc:464] Registering TS with master...
I20260927 13:51:38.859086 371 heartbeater.cc:511] Master 127.20.133.126:40231 requested a full tablet report, sending...
I20260927 13:51:38.860625 32349 ts_manager.cc:194] Registered new tserver with Master: f28051604766451fa742ab8c16c3b164 (127.20.133.67:46615)
I20260927 13:51:38.861455 32349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:50887
I20260927 13:51:38.870797 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:51:38.888608 373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:38.875293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.20.133.126:40231)
0927 13:51:38.875486 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:38.875491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:38.875957 (+ 466us) server_negotiation.cc:407] Connection header received
0927 13:51:38.876122 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:38.876126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:38.876227 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:38.876323 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:51:38.877745 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.878528 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.879820 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.880240 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.880360 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:38.882638 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:51:38.882644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:51:38.882647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:51:38.882684 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:51:38.885247 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:38.885979 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:38.885984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:38.885987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:38.886128 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:38.886730 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:38.886737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:38.886740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:38.887308 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:51:38.887582 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:38.888105 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 13:51:38.888319 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:51:38.893752 32349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:51:38.901265 32349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:51:38.923739 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:38.913841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:46615 (local address 127.0.0.1:56780)
0927 13:51:38.914734 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 13:51:38.914779 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:51:38.914947 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:51:38.916384 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:51:38.916391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:51:38.916419 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:51:38.916828 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:38.916841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.918905 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.918911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:38.921290 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:38.921306 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.922684 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.922691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:38.922909 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:38.922987 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 13:51:38.923127 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 13:51:38.923326 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":55,"thread_start_us":486,"threads_started":1}
I20260927 13:51:38.923710 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:38.914148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.20.133.67:46615)
0927 13:51:38.915211 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:38.915218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:38.915247 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:51:38.915338 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:38.915345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:38.916143 (+ 798us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:38.917033 (+ 890us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:51:38.917289 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.918760 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.921490 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.922488 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.922612 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:38.922833 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:38.923168 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 13:51:38.923356 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":773,"threads_started":1}
I20260927 13:51:38.925203 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:38.915673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:45801 (local address 127.0.0.1:45186)
0927 13:51:38.916176 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 13:51:38.916191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:51:38.916287 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:51:38.917349 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:51:38.917354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:51:38.917380 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:51:38.917744 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:38.917755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.919636 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.919641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:38.922615 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:38.922630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.924437 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.924443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:38.924629 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:38.924666 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 13:51:38.924727 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 13:51:38.924846 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":311,"threads_started":1}
I20260927 13:51:38.925551 386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:38.915119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:41681 (local address 127.0.0.1:40922)
0927 13:51:38.915622 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 13:51:38.915638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:51:38.915738 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:51:38.918550 (+ 2812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:51:38.918555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:51:38.918577 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:51:38.918942 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:38.918953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.920387 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.920392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:38.922952 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:38.922965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.924670 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.924676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:38.924835 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:38.924863 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 13:51:38.924945 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 13:51:38.925069 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":64,"thread_start_us":268,"threads_started":1}
I20260927 13:51:38.926096 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:38.915425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.20.133.66:41681)
0927 13:51:38.918150 (+ 2725us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:38.918155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:38.918174 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:51:38.918241 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:38.918245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:38.918385 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:38.918532 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:51:38.919141 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.920230 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.923105 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.924461 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.924582 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:38.924756 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:38.925447 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 13:51:38.925771 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2602,"thread_start_us":2494,"threads_started":1}
I20260927 13:51:38.926281 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:38.916030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.20.133.65:45801)
0927 13:51:38.916826 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:38.916833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:38.916858 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:51:38.916952 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:38.916959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:38.917155 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:38.917925 (+ 770us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:51:38.918139 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.919498 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.922800 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:38.924204 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:38.925617 (+ 1413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:38.925830 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:38.925887 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 13:51:38.926009 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":475,"threads_started":1}
I20260927 13:51:38.927979 306 tablet_service.cc:1467] Processing CreateTablet for tablet abf8660e578d4cf79b20f513fc6b853b (DEFAULT_TABLE table=test-table [id=342fa02574794be894567193bb5fe6a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:51:38.929701 306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf8660e578d4cf79b20f513fc6b853b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:51:38.929327 32497 tablet_service.cc:1467] Processing CreateTablet for tablet abf8660e578d4cf79b20f513fc6b853b (DEFAULT_TABLE table=test-table [id=342fa02574794be894567193bb5fe6a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:51:38.929327 32640 tablet_service.cc:1467] Processing CreateTablet for tablet abf8660e578d4cf79b20f513fc6b853b (DEFAULT_TABLE table=test-table [id=342fa02574794be894567193bb5fe6a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:51:38.930708 32640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf8660e578d4cf79b20f513fc6b853b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:51:38.930719 32497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf8660e578d4cf79b20f513fc6b853b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:51:38.941962 390 tablet_bootstrap.cc:492] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: Bootstrap starting.
I20260927 13:51:38.945294 390 tablet_bootstrap.cc:654] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: Neither blocks nor log segments found. Creating new log.
I20260927 13:51:38.945914 391 tablet_bootstrap.cc:492] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: Bootstrap starting.
I20260927 13:51:38.946995 392 tablet_bootstrap.cc:492] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: Bootstrap starting.
I20260927 13:51:38.948441 390 log.cc:824] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: Log is configured to *not* fsync() on all Append() calls
I20260927 13:51:38.949345 391 tablet_bootstrap.cc:654] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: Neither blocks nor log segments found. Creating new log.
I20260927 13:51:38.950374 392 tablet_bootstrap.cc:654] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: Neither blocks nor log segments found. Creating new log.
I20260927 13:51:38.950428 391 log.cc:824] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: Log is configured to *not* fsync() on all Append() calls
I20260927 13:51:38.951584 392 log.cc:824] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: Log is configured to *not* fsync() on all Append() calls
I20260927 13:51:38.951709 390 tablet_bootstrap.cc:492] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: No bootstrap required, opened a new log
I20260927 13:51:38.951987 390 ts_tablet_manager.cc:1397] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 13:51:38.953843 391 tablet_bootstrap.cc:492] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: No bootstrap required, opened a new log
I20260927 13:51:38.954147 391 ts_tablet_manager.cc:1397] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260927 13:51:38.954804 392 tablet_bootstrap.cc:492] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: No bootstrap required, opened a new log
I20260927 13:51:38.955078 392 ts_tablet_manager.cc:1397] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 13:51:38.962801 390 raft_consensus.cc:348] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 } } peers { permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 } } peers { permanent_uuid: "476ce715c3714519886633752f9ccb63" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45801 } }
I20260927 13:51:38.963377 390 raft_consensus.cc:374] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:51:38.963490 390 raft_consensus.cc:729] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28051604766451fa742ab8c16c3b164, State: Initialized, Role: FOLLOWER
I20260927 13:51:38.964073 392 raft_consensus.cc:348] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 } } peers { permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 } } peers { permanent_uuid: "476ce715c3714519886633752f9ccb63" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45801 } }
I20260927 13:51:38.964234 390 consensus_queue.cc:260] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 } } peers { permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 } } peers { permanent_uuid: "476ce715c3714519886633752f9ccb63" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45801 } }
I20260927 13:51:38.964568 392 raft_consensus.cc:374] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:51:38.964692 392 raft_consensus.cc:729] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476ce715c3714519886633752f9ccb63, State: Initialized, Role: FOLLOWER
I20260927 13:51:38.964637 391 raft_consensus.cc:348] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 } } peers { permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 } } peers { permanent_uuid: "476ce715c3714519886633752f9ccb63" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45801 } }
I20260927 13:51:38.965066 391 raft_consensus.cc:374] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:51:38.965173 391 raft_consensus.cc:729] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70b5a0610b94f70b6c55281f3346e37, State: Initialized, Role: FOLLOWER
I20260927 13:51:38.965248 392 consensus_queue.cc:260] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 } } peers { permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 } } peers { permanent_uuid: "476ce715c3714519886633752f9ccb63" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45801 } }
I20260927 13:51:38.965894 371 heartbeater.cc:503] Master 127.20.133.126:40231 was elected leader, sending a full tablet report...
I20260927 13:51:38.965873 391 consensus_queue.cc:260] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 } } peers { permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 } } peers { permanent_uuid: "476ce715c3714519886633752f9ccb63" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45801 } }
I20260927 13:51:38.966648 390 ts_tablet_manager.cc:1428] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260927 13:51:38.969673 32705 heartbeater.cc:503] Master 127.20.133.126:40231 was elected leader, sending a full tablet report...
I20260927 13:51:38.969964 391 ts_tablet_manager.cc:1428] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260927 13:51:38.972306 392 ts_tablet_manager.cc:1428] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20260927 13:51:39.086042 372 tablet.cc:2369] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:51:39.101218 32563 tablet.cc:2369] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:51:39.138221 32706 tablet.cc:2369] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:51:39.290709 397 raft_consensus.cc:482] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:51:39.291024 397 raft_consensus.cc:504] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 } } peers { permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 } } peers { permanent_uuid: "476ce715c3714519886633752f9ccb63" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45801 } }
I20260927 13:51:39.292761 397 leader_election.cc:288] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f28051604766451fa742ab8c16c3b164 (127.20.133.67:46615), c70b5a0610b94f70b6c55281f3346e37 (127.20.133.66:41681)
I20260927 13:51:39.300225 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.293307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:57835 (local address 127.20.133.67:46615)
0927 13:51:39.293576 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:39.293584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:39.294244 (+ 660us) server_negotiation.cc:407] Connection header received
0927 13:51:39.294524 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:39.294531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:39.294614 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:39.294718 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:51:39.295574 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.296431 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.298385 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.299239 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.299408 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.299495 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:39.299787 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 13:51:39.299941 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 13:51:39.300325 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.293313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:46615 (local address 127.20.133.65:57835)
0927 13:51:39.294098 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 13:51:39.294118 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:51:39.294309 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:51:39.294889 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:51:39.294894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:51:39.294935 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:51:39.295407 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:39.295421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.296571 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.296577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:39.298211 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:39.298225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.299467 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.299474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:39.299629 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.299656 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 13:51:39.299744 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 13:51:39.299920 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":446,"threads_started":1}
I20260927 13:51:39.300798 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.294251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:41681 (local address 127.20.133.65:56597)
0927 13:51:39.294949 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 13:51:39.294965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:51:39.295141 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:51:39.295589 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:51:39.295594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:51:39.295625 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:51:39.296016 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:39.296030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.297154 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.297158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:39.298705 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:39.298720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.299893 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.299900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:39.300045 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.300075 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:51:39.300168 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 13:51:39.300367 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":360,"threads_started":1}
I20260927 13:51:39.301013 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.294492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:56597 (local address 127.20.133.66:41681)
0927 13:51:39.294762 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:39.294768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:39.295079 (+ 311us) server_negotiation.cc:407] Connection header received
0927 13:51:39.295290 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:39.295295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:39.295369 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:39.295569 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:51:39.296228 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.296992 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.298870 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.299648 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.299813 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.299895 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:39.300235 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 13:51:39.300788 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 13:51:39.301589 326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf8660e578d4cf79b20f513fc6b853b" candidate_uuid: "476ce715c3714519886633752f9ccb63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28051604766451fa742ab8c16c3b164" is_pre_election: true
I20260927 13:51:39.301944 32660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf8660e578d4cf79b20f513fc6b853b" candidate_uuid: "476ce715c3714519886633752f9ccb63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70b5a0610b94f70b6c55281f3346e37" is_pre_election: true
I20260927 13:51:39.302172 326 raft_consensus.cc:2454] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476ce715c3714519886633752f9ccb63 in term 0.
I20260927 13:51:39.302553 32660 raft_consensus.cc:2454] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476ce715c3714519886633752f9ccb63 in term 0.
I20260927 13:51:39.302793 398 raft_consensus.cc:482] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:51:39.302954 32452 leader_election.cc:302] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476ce715c3714519886633752f9ccb63, f28051604766451fa742ab8c16c3b164; no voters:
I20260927 13:51:39.303041 398 raft_consensus.cc:504] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 } } peers { permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 } } peers { permanent_uuid: "476ce715c3714519886633752f9ccb63" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45801 } }
I20260927 13:51:39.303496 397 raft_consensus.cc:2781] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:51:39.303719 397 raft_consensus.cc:482] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:51:39.303833 397 raft_consensus.cc:3037] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:51:39.305408 398 leader_election.cc:288] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f28051604766451fa742ab8c16c3b164 (127.20.133.67:46615), 476ce715c3714519886633752f9ccb63 (127.20.133.65:45801)
I20260927 13:51:39.307432 397 raft_consensus.cc:504] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 } } peers { permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 } } peers { permanent_uuid: "476ce715c3714519886633752f9ccb63" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45801 } }
I20260927 13:51:39.308307 397 leader_election.cc:288] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [CANDIDATE]: Term 1 election: Requested vote from peers f28051604766451fa742ab8c16c3b164 (127.20.133.67:46615), c70b5a0610b94f70b6c55281f3346e37 (127.20.133.66:41681)
I20260927 13:51:39.309026 32660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf8660e578d4cf79b20f513fc6b853b" candidate_uuid: "476ce715c3714519886633752f9ccb63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70b5a0610b94f70b6c55281f3346e37"
I20260927 13:51:39.309075 326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf8660e578d4cf79b20f513fc6b853b" candidate_uuid: "476ce715c3714519886633752f9ccb63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28051604766451fa742ab8c16c3b164"
I20260927 13:51:39.309286 32660 raft_consensus.cc:3037] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:51:39.309372 326 raft_consensus.cc:3037] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:51:39.312479 326 raft_consensus.cc:2454] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476ce715c3714519886633752f9ccb63 in term 1.
I20260927 13:51:39.312523 32660 raft_consensus.cc:2454] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476ce715c3714519886633752f9ccb63 in term 1.
I20260927 13:51:39.313243 32452 leader_election.cc:302] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476ce715c3714519886633752f9ccb63, c70b5a0610b94f70b6c55281f3346e37; no voters:
I20260927 13:51:39.313602 397 raft_consensus.cc:2781] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:51:39.313953 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.305725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:46615 (local address 127.20.133.66:32979)
0927 13:51:39.306373 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 13:51:39.306395 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:51:39.306648 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:51:39.307178 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:51:39.307183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:51:39.307211 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:51:39.307598 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:39.307611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.309102 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.309108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:39.311344 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:39.311361 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.313096 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.313103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:39.313303 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.313342 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 13:51:39.313435 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 13:51:39.313582 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":287,"threads_started":1}
I20260927 13:51:39.314834 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.305798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:32979 (local address 127.20.133.67:46615)
0927 13:51:39.306031 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:39.306039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:39.306511 (+ 472us) server_negotiation.cc:407] Connection header received
0927 13:51:39.306817 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:39.306824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:39.306901 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:39.306960 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:51:39.307772 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.308926 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.311524 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.312873 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.313034 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.313124 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:39.314333 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0927 13:51:39.314542 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:51:39.315407 397 raft_consensus.cc:686] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 1 LEADER]: Becoming Leader. State: Replica: 476ce715c3714519886633752f9ccb63, State: Running, Role: LEADER
I20260927 13:51:39.315474 326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf8660e578d4cf79b20f513fc6b853b" candidate_uuid: "c70b5a0610b94f70b6c55281f3346e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28051604766451fa742ab8c16c3b164" is_pre_election: true
I20260927 13:51:39.315871 326 raft_consensus.cc:2379] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c70b5a0610b94f70b6c55281f3346e37 in current term 1: Already voted for candidate 476ce715c3714519886633752f9ccb63 in this term.
I20260927 13:51:39.316305 397 consensus_queue.cc:237] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 } } peers { permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 } } peers { permanent_uuid: "476ce715c3714519886633752f9ccb63" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45801 } }
I20260927 13:51:39.319674 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.306448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:45801 (local address 127.20.133.66:33565)
0927 13:51:39.307247 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 13:51:39.307263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:51:39.307442 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:51:39.308061 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:51:39.308064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:51:39.308086 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:51:39.308621 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:39.308637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.313879 (+ 5242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.313886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:39.317094 (+ 3208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:51:39.317114 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.318869 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.318876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:51:39.319059 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.319095 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 13:51:39.319190 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 13:51:39.319338 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":472,"threads_started":1}
I20260927 13:51:39.319716 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.306615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:33565 (local address 127.20.133.65:45801)
0927 13:51:39.306809 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:39.306815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:39.307360 (+ 545us) server_negotiation.cc:407] Connection header received
0927 13:51:39.307577 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:39.307582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:39.307650 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:39.307726 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:51:39.308905 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.309985 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.317293 (+ 7308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.318614 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.318815 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.318892 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:39.319235 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 13:51:39.319398 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 13:51:39.320919 32517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf8660e578d4cf79b20f513fc6b853b" candidate_uuid: "c70b5a0610b94f70b6c55281f3346e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476ce715c3714519886633752f9ccb63" is_pre_election: true
I20260927 13:51:39.322265 32595 leader_election.cc:302] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c70b5a0610b94f70b6c55281f3346e37; no voters: 476ce715c3714519886633752f9ccb63, f28051604766451fa742ab8c16c3b164
I20260927 13:51:39.322849 398 raft_consensus.cc:2726] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 13:51:39.323539 32348 catalog_manager.cc:5495] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 476ce715c3714519886633752f9ccb63 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "476ce715c3714519886633752f9ccb63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476ce715c3714519886633752f9ccb63" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45801 } health_report { overall_health: HEALTHY } } }
I20260927 13:51:39.393458 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.386683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45200 (local address 127.20.133.65:45801)
0927 13:51:39.386970 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:39.386978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:39.387170 (+ 192us) server_negotiation.cc:407] Connection header received
0927 13:51:39.387478 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:39.387483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:39.387604 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:39.387775 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:51:39.388232 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.389168 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.390686 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.391141 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.391258 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.391881 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:51:39.392681 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:51:39.392840 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:39.392978 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 13:51:39.393091 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 13:51:39.404423 32660 raft_consensus.cc:1261] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 1 FOLLOWER]: Refusing update from remote peer 476ce715c3714519886633752f9ccb63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:51:39.404485 326 raft_consensus.cc:1261] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164 [term 1 FOLLOWER]: Refusing update from remote peer 476ce715c3714519886633752f9ccb63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:51:39.405448 397 consensus_queue.cc:1035] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c70b5a0610b94f70b6c55281f3346e37" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 41681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:51:39.405809 403 consensus_queue.cc:1035] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28051604766451fa742ab8c16c3b164" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:51:39.436164 32347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:51:39.436538 32347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:51:39.440811 32347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb03f4b5730d463ca6d3ec57b50f5599: Sending DeleteTablet for 3 replicas of tablet abf8660e578d4cf79b20f513fc6b853b
I20260927 13:51:39.442480 306 tablet_service.cc:1514] Processing DeleteTablet for tablet abf8660e578d4cf79b20f513fc6b853b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:51:39 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:56780
I20260927 13:51:39.442559 32640 tablet_service.cc:1514] Processing DeleteTablet for tablet abf8660e578d4cf79b20f513fc6b853b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:51:39 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:40922
I20260927 13:51:39.443390 32497 tablet_service.cc:1514] Processing DeleteTablet for tablet abf8660e578d4cf79b20f513fc6b853b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:51:39 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:45186
I20260927 13:51:39.444446 417 tablet_replica.cc:330] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: stopping tablet replica
I20260927 13:51:39.445078 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 32429
I20260927 13:51:39.445055 417 raft_consensus.cc:2229] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:51:39.445621 417 raft_consensus.cc:2258] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:51:39.454993 416 tablet_replica.cc:330] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: stopping tablet replica
I20260927 13:51:39.455793 416 raft_consensus.cc:2229] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:51:39.456621 416 raft_consensus.cc:2258] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:51:39.456975 417 ts_tablet_manager.cc:1905] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:51:39.457743 418 tablet_replica.cc:330] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: stopping tablet replica
I20260927 13:51:39.458398 418 raft_consensus.cc:2229] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:51:39.459288 416 ts_tablet_manager.cc:1905] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:51:39.459283 418 raft_consensus.cc:2258] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:51:39.462213 418 ts_tablet_manager.cc:1905] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:51:39.465133 417 ts_tablet_manager.cc:1918] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 27 13:51:39 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:51:39.465401 417 log.cc:1196] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/abf8660e578d4cf79b20f513fc6b853b
I20260927 13:51:39.465987 417 ts_tablet_manager.cc:1939] T abf8660e578d4cf79b20f513fc6b853b P f28051604766451fa742ab8c16c3b164: Deleting consensus metadata
I20260927 13:51:39.466647 416 ts_tablet_manager.cc:1918] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:51:39.466815 416 log.cc:1196] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/abf8660e578d4cf79b20f513fc6b853b
I20260927 13:51:39.467612 416 ts_tablet_manager.cc:1939] T abf8660e578d4cf79b20f513fc6b853b P c70b5a0610b94f70b6c55281f3346e37: Deleting consensus metadata
I20260927 13:51:39.468325 32335 catalog_manager.cc:4841] TS f28051604766451fa742ab8c16c3b164 (127.20.133.67:46615): tablet abf8660e578d4cf79b20f513fc6b853b (table test-table [id=342fa02574794be894567193bb5fe6a2]) successfully deleted
I20260927 13:51:39.469556 32335 catalog_manager.cc:4841] TS c70b5a0610b94f70b6c55281f3346e37 (127.20.133.66:41681): tablet abf8660e578d4cf79b20f513fc6b853b (table test-table [id=342fa02574794be894567193bb5fe6a2]) successfully deleted
I20260927 13:51:39.473052 418 ts_tablet_manager.cc:1918] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:51:39.473322 418 log.cc:1196] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/abf8660e578d4cf79b20f513fc6b853b
I20260927 13:51:39.473974 418 ts_tablet_manager.cc:1939] T abf8660e578d4cf79b20f513fc6b853b P 476ce715c3714519886633752f9ccb63: Deleting consensus metadata
I20260927 13:51:39.476377 32335 catalog_manager.cc:4841] TS 476ce715c3714519886633752f9ccb63 (127.20.133.65:45801): tablet abf8660e578d4cf79b20f513fc6b853b (table test-table [id=342fa02574794be894567193bb5fe6a2]) successfully deleted
I20260927 13:51:39.476991 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.447558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.20.133.65:45801)
0927 13:51:39.447767 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:39.447774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:39.448235 (+ 461us) server_negotiation.cc:407] Connection header received
0927 13:51:39.448604 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:39.448612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:39.448719 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:39.448814 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:51:39.451056 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.452534 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.455384 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.456063 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.456189 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.469882 (+ 13693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:51:39.469906 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:51:39.469931 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:51:39.469992 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:51:39.473803 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:39.474667 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:39.474675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:39.474677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:39.474808 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:39.475191 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:39.475196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:39.475199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:39.475707 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:51:39.476045 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:39.476466 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 13:51:39.476728 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:51:39.478536 32557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:51:39.588652 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 32429
I20260927 13:51:39.605530 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 32568
Sep 27 13:51:39 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:51:39.622213 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.606413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.20.133.66:41681)
0927 13:51:39.606585 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:39.606590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:39.606779 (+ 189us) server_negotiation.cc:407] Connection header received
0927 13:51:39.606951 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:39.606954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:39.607023 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:39.607139 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:51:39.608656 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.609366 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.610726 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.611119 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.611215 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.616306 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:51:39.616322 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:51:39.616380 (+ 58us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:51:39.616433 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:51:39.619188 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:39.619997 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:39.620004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:39.620007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:39.620162 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:39.620631 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:39.620641 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:39.620643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:39.621041 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:51:39.621271 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:39.621740 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 13:51:39.621925 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:51:39.623066 32700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:51:39.715504 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 32568
I20260927 13:51:39.729043 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 32711
Sep 27 13:51:39 dist-test-slave-q25r krb5kdc[31873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:51:39.745332 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:51:39.729910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.20.133.67:46615)
0927 13:51:39.730136 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 13:51:39.730143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:51:39.730165 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:51:39.730325 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:51:39.730329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:51:39.730413 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:51:39.730511 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:51:39.731979 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.732771 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.734293 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:51:39.734762 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:51:39.734872 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:51:39.739767 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:51:39.739780 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:51:39.739796 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:51:39.739835 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:51:39.742600 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:39.743488 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:39.743495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:39.743497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:39.743622 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:51:39.743970 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:51:39.743973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:51:39.743975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:51:39.744304 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:51:39.744591 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 13:51:39.744907 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 13:51:39.745071 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 13:51:39.746150 366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:51:39.842921 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 32711
I20260927 13:51:39.858038 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 32317
I20260927 13:51:39.859618 32379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:51:39.973850 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 32317
I20260927 13:51:39.996148 21013 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 13:51:40.333540 21013 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 13:51:40.333824 21013 mini_ranger.cc:66] Stopping Ranger...
I20260927 13:51:40.680626 21013 mini_ranger.cc:68] Stopped Ranger
2026-09-27 13:51:40.681 UTC [31903] LOG: received smart shutdown request
2026-09-27 13:51:40.685 UTC [31903] LOG: background worker "logical replication launcher" (PID 31912) exited with exit code 1
2026-09-27 13:51:40.685 UTC [31907] LOG: shutting down
2026-09-27 13:51:40.857 UTC [31903] LOG: database system is shut down
2026-09-27T13:51:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51956 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:51:41 dist-test-slave-q25r krb5kdc[432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:51:41 dist-test-slave-q25r krb5kdc[432](info): set up 2 sockets
Sep 27 13:51:41 dist-test-slave-q25r krb5kdc[432](info): commencing operation
krb5kdc: starting...
W20260927 13:51:42.032609 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:51:42 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:51:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.133.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.133.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/rangeradmin_127.20.133.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/rangeradmin_127.20.133.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.133.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.133.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/rangerlookup_127.20.133.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/rangerlookup_127.20.133.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.133.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.133.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/HTTP_127.20.133.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/HTTP_127.20.133.84@KRBTEST.COM.keytab.
I20260927 13:51:42.407167 21013 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskM7FxcK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/postgres -l logfile start
2026-09-27 13:51:45.006 UTC [462] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 13:51:45.006 UTC [462] LOG: listening on IPv4 address "127.20.133.84", port 39859
2026-09-27 13:51:45.012 UTC [462] LOG: listening on Unix socket "/tmp/.s.PGSQL.39859"
2026-09-27 13:51:45.039 UTC [465] LOG: database system was shut down at 2026-09-27 13:51:43 UTC
2026-09-27 13:51:45.043 UTC [462] LOG: database system is ready to accept connections
I20260927 13:51:45.944257 21013 mini_postgres.cc:96] Postgres bound to 39859
2026-09-27 13:51:45.964 UTC [473] FATAL: database "slave" does not exist
127.20.133.84:39859 - accepting connections
I20260927 13:51:45.965561 21013 mini_ranger.cc:161] Starting Ranger...
I20260927 13:51:45.995216 21013 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 13:51:46.482569 21013 mini_ranger.cc:87] Created ranger Postgres database
I20260927 13:51:46.482698 21013 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-admin
2026-09-27 13:51:46,795 [I] DB FLAVOR :POSTGRES
2026-09-27 13:51:46,795 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:51:46,795 [I] Checking connection..
2026-09-27 13:51:46,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 13:51:47,057 [I] Checking connection passed.
2026-09-27 13:51:47,057 [I] --------- Verifying version history table ---------
2026-09-27 13:51:47,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:51:47,317 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:51:47,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:51:47,579 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:51:47,579 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 13:51:47,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 13:51:47.837 UTC [537] WARNING: there is no transaction in progress
2026-09-27 13:51:47,843 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 13:51:47,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:51:48,088 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 13:51:48,088 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:51:48,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 13:51:48,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 13:51:48,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q25r.c.gcp-upstream.internal','N') ;"
2026-09-27 13:51:48,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:51:49,072 [I] Table x_portal_user does not exist in database ranger
2026-09-27 13:51:49,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:51:49,309 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 13:51:49,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:51:49,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:51:49,811 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 13:51:49,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 13:51:50.865 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:50.884 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:50.904 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:50.922 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:50.940 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:50.963 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:51.025 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:51.032 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:51.047 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:51.060 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:51.071 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:51.087 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:51.099 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:51.107 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:51.115 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:51.123 UTC [645] WARNING: there is no transaction in progress
2026-09-27 13:51:51,843 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 13:51:51,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 13:51:52,124 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 13:51:52,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:51:52,398 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 13:51:52,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:51:52,670 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 13:51:52,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:51:52,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:51:53,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q25r.c.gcp-upstream.internal';"
2026-09-27 13:51:53,484 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 13:51:53,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 13:51:53,731 [I] DB_PATCHES have already been applied
I20260927 13:51:53.737129 21013 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-admin:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskM7FxcK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 13:51:53.737215 21013 mini_ranger.cc:193] Using host: 127.20.133.84
I20260927 13:51:53.751363 21013 mini_ranger.cc:227] Ranger admin URL: http://127.20.133.84:47479
Sep 27, 2026 1:51:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:51:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:51:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 1:51:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 1:51:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 1:51:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 1:51:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 13:51:54.822477 21013 mini_ranger.cc:160] Time spent starting Ranger: real 8.857s user 0.000s sys 0.044s
Sep 27, 2026 1:52:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 1:52:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 1:52:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 1:52:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 1:52:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 13:52:16.646335 21013 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.133.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.133.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/rangerkms_127.20.133.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.133.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/rangerkms_127.20.133.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.133.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.133.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.133.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/HTTP_127.20.133.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.133.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/HTTP_127.20.133.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 13:52:16 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517136, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 13:52:16.834438 21013 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 13:52:16.857457 21013 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 13:52:17.726763 21013 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 13:52:17.726866 21013 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-kms
I20260927 13:52:17.726905 21013 mini_ranger_kms.cc:227] Using postgres at 127.20.133.84:39859
2026-09-27 13:52:18,058 [I] DB FLAVOR :POSTGRES
2026-09-27 13:52:18,059 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:52:18,059 [I] Checking connection
2026-09-27 13:52:18,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 13:52:18,295 [I] connection success
2026-09-27 13:52:18,295 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 13:52:18,295 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 13:52:18,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 13:52:18,552 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 13:52:18,552 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:52:18,552 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 13:52:18,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.133.84:39859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 13:52:18,815 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 13:52:19.287058 21013 mini_ranger_kms.cc:310] Created kms service
I20260927 13:52:19.463711 21013 mini_ranger_kms.cc:326] Created kudu user
I20260927 13:52:19.509545 21013 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 13:52:19 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517136, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:52:19.894913 21013 mini_ranger_kms.cc:384] Added ranger policy
I20260927 13:52:19.895223 21013 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-kms:/tmp/dist-test-taskM7FxcK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskM7FxcK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 13:52:19.895280 21013 mini_ranger_kms.cc:240] Using host: 127.20.133.84
I20260927 13:52:19.909617 21013 mini_ranger_kms.cc:276] Ranger KMS PID: 825
I20260927 13:52:19.909751 21013 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.133.84:42219
Sep 27, 2026 1:52:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:52:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 1:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 1:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 1:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 13:52:21.999976 21013 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.166s user 0.087s sys 0.267s
I20260927 13:52:22.000231 21013 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 13:52:22.000510 21013 mini_ranger_kms.cc:401] 127.20.133.84:42219/kms/v1/keys
Sep 27 13:52:22 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790517142, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.133.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 13:52:22 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517142, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.133.84@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
Sep 27 13:52:23 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517142, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.133.84@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
Sep 27 13:52:25 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:52:25.464766 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41653
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:41653
--ranger_config_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:42219/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:52:25.808001 876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:25.808327 876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:25.808490 876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:25.816115 876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:52:25.816275 876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:25.816375 876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:25.816453 876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:52:25.816530 876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:52:25.827262 876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.133.84:42219/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:41653
--ranger_config_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:25.828679 876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:25.830106 876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:25.841995 882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:25.842232 884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:25.841974 881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:25.843137 876 server_base.cc:1034] running on GCE node
I20260927 13:52:25.843946 876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:25.845537 876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:25.846784 876 hybrid_clock.cc:648] HybridClock initialized: now 1790517145846731 us; error 47 us; skew 500 ppm
I20260927 13:52:25.847154 876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:25 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517145, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:25.850688 876 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:52:25.852694 876 webserver.cc:460] Webserver started at http://127.20.133.126:35765/ using document root <none> and password file <none>
I20260927 13:52:25.853430 876 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:25.853549 876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:25.853876 876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:52:25 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517145, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:52:25.915311 876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "cd60d9e46b214814945033fcb3821ff9"
format_stamp: "Formatted at 2026-09-27 13:52:25 on dist-test-slave-q25r"
server_key: "809bf5ac0deb772ebd02eac8c704c9d8"
server_key_iv: "33809b55388746a3a00032cc848272b7"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:25.916518 876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "cd60d9e46b214814945033fcb3821ff9"
format_stamp: "Formatted at 2026-09-27 13:52:25 on dist-test-slave-q25r"
server_key: "809bf5ac0deb772ebd02eac8c704c9d8"
server_key_iv: "33809b55388746a3a00032cc848272b7"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:25.924201 876 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 13:52:25.942579 891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:25.944082 876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:52:25.944263 876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "cd60d9e46b214814945033fcb3821ff9"
format_stamp: "Formatted at 2026-09-27 13:52:25 on dist-test-slave-q25r"
server_key: "809bf5ac0deb772ebd02eac8c704c9d8"
server_key_iv: "33809b55388746a3a00032cc848272b7"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:25.944535 876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:25.972517 876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:25.976946 876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:25.977273 876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:25.996766 876 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:41653
I20260927 13:52:25.996766 942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:41653 every 8 connection(s)
I20260927 13:52:25.998242 876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:52:26.002728 943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:26.002907 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 876
I20260927 13:52:26.003115 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:52:26.003973 21013 external_mini_cluster.cc:1428] Setting key aab1df8627c15d049728c0e2ed2ee3f2
I20260927 13:52:26.020440 943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9: Bootstrap starting.
I20260927 13:52:26.025094 943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:26.026343 943 log.cc:824] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:26.029601 943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9: No bootstrap required, opened a new log
Sep 27 13:52:26 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:26.040007 943 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } }
I20260927 13:52:26.040530 943 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:26.040752 943 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd60d9e46b214814945033fcb3821ff9, State: Initialized, Role: FOLLOWER
I20260927 13:52:26.041546 943 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } }
I20260927 13:52:26.041818 943 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:52:26.041971 943 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:52:26.042207 943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:26.045738 943 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } }
I20260927 13:52:26.046579 943 leader_election.cc:302] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd60d9e46b214814945033fcb3821ff9; no voters:
I20260927 13:52:26.047262 943 leader_election.cc:288] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:52:26.047385 948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:52:26.048149 946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:26.006248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.20.133.126:41653)
0927 13:52:26.021096 (+ 14848us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:26.021115 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:26.021179 (+ 64us) server_negotiation.cc:407] Connection header received
0927 13:52:26.023790 (+ 2611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:26.023823 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:26.024431 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:26.024917 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:26.026612 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:26.028780 (+ 2168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:26.030764 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:26.031853 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:26.032049 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:26.037669 (+ 5620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:26.037686 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:26.037710 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:26.037778 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:26.041882 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:26.043234 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:26.043251 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:26.043273 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:26.043553 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:26.044229 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:26.044235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:26.044238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:26.045255 (+ 1017us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:26.045583 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:26.046151 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 13:52:26.046622 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14556,"thread_start_us":15784,"threads_started":1}
I20260927 13:52:26.054527 948 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 1 LEADER]: Becoming Leader. State: Replica: cd60d9e46b214814945033fcb3821ff9, State: Running, Role: LEADER
I20260927 13:52:26.056113 943 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:52:26.056001 948 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } }
I20260927 13:52:26.064471 950 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd60d9e46b214814945033fcb3821ff9. Latest consensus state: current_term: 1 leader_uuid: "cd60d9e46b214814945033fcb3821ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } } }
I20260927 13:52:26.064965 950 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:26.065547 949 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd60d9e46b214814945033fcb3821ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } } }
I20260927 13:52:26.065820 949 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:26.074505 955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:52:26.082581 955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:52:26.109908 955 catalog_manager.cc:1347] Generated new cluster ID: 545f9d6bf5b24ee18d99f7a54ba4fc80
I20260927 13:52:26.110046 955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:52:26.152678 955 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:52:26.154663 955 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:52:26.174114 943 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 13:52:26.175813 955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9: Generated new TSK 0
I20260927 13:52:26.176942 955 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:52:27.036936 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:42219/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:41653
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:52:27 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790517147, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 13:52:27.478466 989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:27.478901 989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:27.479120 989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:27.491336 989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:27.491575 989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:27.491793 989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:52:27.508157 989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.133.84:42219/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:41653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:27.510079 989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:27.511914 989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:27.530413 999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:27.530706 1000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:27.531533 1002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:27.532320 989 server_base.cc:1034] running on GCE node
I20260927 13:52:27.532974 989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:27.533977 989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:27.535221 989 hybrid_clock.cc:648] HybridClock initialized: now 1790517147535154 us; error 51 us; skew 500 ppm
I20260927 13:52:27.535573 989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:27 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517147, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:27.538975 989 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:52:27.540872 989 webserver.cc:460] Webserver started at http://127.20.133.65:34205/ using document root <none> and password file <none>
I20260927 13:52:27.541821 989 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:27.541966 989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:27.542335 989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:52:27 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517147, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:52:27.564901 989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "f4815101859f427c80ccf7bc5f6618ff"
format_stamp: "Formatted at 2026-09-27 13:52:27 on dist-test-slave-q25r"
server_key: "ab1f6f77229b5ae4877062a379b0f9e2"
server_key_iv: "e6f47a3fb883a748e92e630feb497cf9"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:27.566170 989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "f4815101859f427c80ccf7bc5f6618ff"
format_stamp: "Formatted at 2026-09-27 13:52:27 on dist-test-slave-q25r"
server_key: "ab1f6f77229b5ae4877062a379b0f9e2"
server_key_iv: "e6f47a3fb883a748e92e630feb497cf9"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:27.578325 989 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.008s
I20260927 13:52:27.610366 1009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:27.613005 989 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260927 13:52:27.613315 989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "f4815101859f427c80ccf7bc5f6618ff"
format_stamp: "Formatted at 2026-09-27 13:52:27 on dist-test-slave-q25r"
server_key: "ab1f6f77229b5ae4877062a379b0f9e2"
server_key_iv: "e6f47a3fb883a748e92e630feb497cf9"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:27.613672 989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:27.662922 989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:27.668767 989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:27.669219 989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:27.670444 989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:27.672130 989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:27.672219 989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:27.672394 989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:27.672451 989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:52:27 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517147, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:52:27.852790 989 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:43859
I20260927 13:52:27.852806 1121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:43859 every 8 connection(s)
I20260927 13:52:27.854453 989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:52:27.857587 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 989
I20260927 13:52:27.857820 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:52:27.858668 21013 external_mini_cluster.cc:1428] Setting key 8135455d08b170cead5a4889539ad3c8
Sep 27 13:52:27 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517147, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:27.947563 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:27.895645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:41653 (local address 127.20.133.65:44419)
0927 13:52:27.910602 (+ 14957us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:27.910663 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:27.917692 (+ 7029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:27.925352 (+ 7660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:27.925407 (+ 55us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:27.927466 (+ 2059us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:27.928428 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:27.928465 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:27.930194 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:27.930200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:27.932188 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:27.932208 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:27.933605 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:27.933613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:27.933933 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:27.934820 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:27.934855 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:27.937698 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:27.942456 (+ 4758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:27.942468 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:27.942489 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:27.943143 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:27.943626 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:27.943631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:27.943634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:27.943902 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:27.944676 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:27.944685 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:27.945161 (+ 476us) client_negotiation.cc:769] Sending connection context
0927 13:52:27.945473 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 13:52:27.946074 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14598,"thread_start_us":16574,"threads_started":1}
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
I20260927 13:52:27.949116 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:27.895040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:44419 (local address 127.20.133.126:41653)
0927 13:52:27.908861 (+ 13821us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:27.908869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:27.910842 (+ 1973us) server_negotiation.cc:407] Connection header received
0927 13:52:27.923081 (+ 12239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:27.923089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:27.923249 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:27.923349 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:27.928743 (+ 5394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:27.929920 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:27.932424 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:27.933089 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:27.933198 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:27.938043 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:27.938063 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:27.938069 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:27.938115 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:27.942223 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:27.943328 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:27.943334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:27.943338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:27.943494 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:27.944040 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:27.944045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:27.944059 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:27.944452 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:27.944602 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:27.948250 (+ 3648us) server_negotiation.cc:299] Negotiation successful
0927 13:52:27.948610 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13614,"thread_start_us":15829,"threads_started":1}
I20260927 13:52:27.951670 1122 heartbeater.cc:347] Connected to a master server at 127.20.133.126:41653
I20260927 13:52:27.952021 1122 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:27.953095 1122 heartbeater.cc:511] Master 127.20.133.126:41653 requested a full tablet report, sending...
I20260927 13:52:27.956248 908 ts_manager.cc:194] Registered new tserver with Master: f4815101859f427c80ccf7bc5f6618ff (127.20.133.65:43859)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
I20260927 13:52:27.968655 908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:44419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
W20260927 13:52:28.010028 21013 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.20.133.66: real 0.148s user 0.008s sys 0.010s
I20260927 13:52:28.010890 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:42219/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:41653
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:52:28 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517147, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
W20260927 13:52:28.549242 1128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:28.549644 1128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:28.549818 1128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:28.561336 1128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:28.561512 1128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:28.561721 1128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:52:28.577174 1128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.133.84:42219/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:41653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:28.578795 1128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:28.580387 1128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:28.597026 1142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:28.597020 1143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:28.598232 1145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:28.598430 1128 server_base.cc:1034] running on GCE node
I20260927 13:52:28.599318 1128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:28.600643 1128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:28.601888 1128 hybrid_clock.cc:648] HybridClock initialized: now 1790517148601823 us; error 54 us; skew 500 ppm
I20260927 13:52:28.602258 1128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:28 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517148, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:28.606426 1128 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:52:28.608234 1128 webserver.cc:460] Webserver started at http://127.20.133.66:43811/ using document root <none> and password file <none>
I20260927 13:52:28.609165 1128 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:28.609305 1128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:28.609656 1128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:52:28 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517148, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:52:28.628196 1128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "a423e7d653fe43eab1a8e28306c6a4f0"
format_stamp: "Formatted at 2026-09-27 13:52:28 on dist-test-slave-q25r"
server_key: "4e43288050e58e94c47d4b11381de0a9"
server_key_iv: "ea43e51a002b3d10da0f49ce0968d1d7"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:28.629415 1128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "a423e7d653fe43eab1a8e28306c6a4f0"
format_stamp: "Formatted at 2026-09-27 13:52:28 on dist-test-slave-q25r"
server_key: "4e43288050e58e94c47d4b11381de0a9"
server_key_iv: "ea43e51a002b3d10da0f49ce0968d1d7"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:28.642040 1128 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.005s
I20260927 13:52:28.664697 1152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:28.666829 1128 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.007s sys 0.000s
I20260927 13:52:28.667078 1128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "a423e7d653fe43eab1a8e28306c6a4f0"
format_stamp: "Formatted at 2026-09-27 13:52:28 on dist-test-slave-q25r"
server_key: "4e43288050e58e94c47d4b11381de0a9"
server_key_iv: "ea43e51a002b3d10da0f49ce0968d1d7"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:28.667380 1128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:28.694628 1128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:28.698642 1128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:28.698933 1128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:28.700124 1128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:28.701720 1128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:28.701807 1128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:28.701926 1128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:28.701989 1128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:28.738641 1128 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:39053
I20260927 13:52:28.738656 1264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:39053 every 8 connection(s)
I20260927 13:52:28.739959 1128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:52:28.741263 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 1128
I20260927 13:52:28.741433 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:52:28.741976 21013 external_mini_cluster.cc:1428] Setting key 646902aa7acfa4beee57613b1237ca83
Sep 27 13:52:28 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517148, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
I20260927 13:52:28.777122 1267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:28.743725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:58659 (local address 127.20.133.126:41653)
0927 13:52:28.759010 (+ 15285us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:28.759016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:28.759050 (+ 34us) server_negotiation.cc:407] Connection header received
0927 13:52:28.761811 (+ 2761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:28.761818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:28.761909 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:28.762007 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:28.763567 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:28.764324 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:28.765989 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:28.766413 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:28.766534 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:28.769883 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:28.769888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:28.769891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:28.769928 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:28.772720 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:28.773744 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:28.773749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:28.773751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:28.773882 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:28.774378 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:28.774382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:28.774385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:28.774686 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:28.774832 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:28.776529 (+ 1697us) server_negotiation.cc:299] Negotiation successful
0927 13:52:28.776769 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15144,"thread_start_us":15321,"threads_started":1}
I20260927 13:52:28.777452 1268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:28.744330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:41653 (local address 127.20.133.66:58659)
0927 13:52:28.758828 (+ 14498us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:28.758891 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:28.761489 (+ 2598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:28.762287 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:28.762297 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:28.762740 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:28.763361 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:28.763380 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:28.764496 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:28.764502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:28.765843 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:28.765856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:28.766648 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:28.766656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:28.766882 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:28.767543 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:28.767571 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:28.769625 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:28.772921 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:28.772930 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:28.772951 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:28.773615 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:28.774014 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:28.774019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:28.774023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:28.774270 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:28.774813 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:28.774821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:28.775200 (+ 379us) client_negotiation.cc:769] Sending connection context
0927 13:52:28.775476 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 13:52:28.775972 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14061,"thread_start_us":14653,"threads_started":1}
I20260927 13:52:28.778946 1265 heartbeater.cc:347] Connected to a master server at 127.20.133.126:41653
I20260927 13:52:28.779247 1265 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:28.779929 1265 heartbeater.cc:511] Master 127.20.133.126:41653 requested a full tablet report, sending...
I20260927 13:52:28.781363 908 ts_manager.cc:194] Registered new tserver with Master: a423e7d653fe43eab1a8e28306c6a4f0 (127.20.133.66:39053)
I20260927 13:52:28.782296 908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:58659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:52:28.803987 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:42219/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:41653
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:52:28.973268 1122 heartbeater.cc:503] Master 127.20.133.126:41653 was elected leader, sending a full tablet report...
W20260927 13:52:29.138181 1271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:29.138485 1271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:29.138625 1271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:29.146114 1271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:29.146232 1271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:29.146373 1271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:52:29.156422 1271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.133.84:42219/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:41653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:29.157792 1271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:29.159065 1271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:29.172079 1279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:29.172164 1276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:29.172144 1277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:29.173022 1271 server_base.cc:1034] running on GCE node
I20260927 13:52:29.173674 1271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:29.174618 1271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:29.175845 1271 hybrid_clock.cc:648] HybridClock initialized: now 1790517149175779 us; error 51 us; skew 500 ppm
I20260927 13:52:29.176179 1271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:29 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:29.179292 1271 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:52:29.180786 1271 webserver.cc:460] Webserver started at http://127.20.133.67:39387/ using document root <none> and password file <none>
I20260927 13:52:29.181460 1271 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:29.181586 1271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:29.181867 1271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:52:29 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:52:29.196640 1271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "174e5ef292354ccc985b1b240e78ba26"
format_stamp: "Formatted at 2026-09-27 13:52:29 on dist-test-slave-q25r"
server_key: "52abf76e02a1fd2d892383e28c860029"
server_key_iv: "326834b0edfc9f205bf36937616c773a"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:29.197576 1271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "174e5ef292354ccc985b1b240e78ba26"
format_stamp: "Formatted at 2026-09-27 13:52:29 on dist-test-slave-q25r"
server_key: "52abf76e02a1fd2d892383e28c860029"
server_key_iv: "326834b0edfc9f205bf36937616c773a"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:29.203125 1271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:52:29.220690 1286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:29.222420 1271 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:52:29.222617 1271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "174e5ef292354ccc985b1b240e78ba26"
format_stamp: "Formatted at 2026-09-27 13:52:29 on dist-test-slave-q25r"
server_key: "52abf76e02a1fd2d892383e28c860029"
server_key_iv: "326834b0edfc9f205bf36937616c773a"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:29.222855 1271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:29.242276 1271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:29.246395 1271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:29.246675 1271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:29.247668 1271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:29.249169 1271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:29.249240 1271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:29.249352 1271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:29.249398 1271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:29.280645 1271 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:39645
I20260927 13:52:29.280706 1398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:39645 every 8 connection(s)
I20260927 13:52:29.281921 1271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:52:29.290681 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 1271
I20260927 13:52:29.290872 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:52:29.291460 21013 external_mini_cluster.cc:1428] Setting key 7881dd44288bd707a309a9c8a6ac2a03
Sep 27 13:52:29 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:29.302006 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:29.284454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:33391 (local address 127.20.133.126:41653)
0927 13:52:29.284975 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:29.284981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:29.285847 (+ 866us) server_negotiation.cc:407] Connection header received
0927 13:52:29.287669 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:29.287674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:29.287753 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:29.287855 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:29.289450 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:29.290207 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:29.292411 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:29.292859 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:29.292968 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:29.296231 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:29.296236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:29.296239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:29.296277 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:29.298903 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:29.299657 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:29.299662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:29.299666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:29.299799 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:29.300230 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:29.300234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:29.300236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:29.300552 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:29.300686 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:29.301414 (+ 728us) server_negotiation.cc:299] Negotiation successful
0927 13:52:29.301649 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":228,"threads_started":1}
I20260927 13:52:29.302971 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:29.284943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:41653 (local address 127.20.133.67:33391)
0927 13:52:29.285685 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:29.285722 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:29.287336 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:29.288126 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:29.288135 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:29.288641 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:29.289242 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:29.289260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:29.290403 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:29.290408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:29.292235 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:29.292248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:29.293082 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:29.293089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:29.293311 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:29.293952 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:29.293976 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:29.296011 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:29.299052 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:29.299063 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:29.299082 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:29.299537 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:29.299906 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:29.299911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:29.299915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:29.300119 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:29.300674 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:29.300682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:29.301040 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 13:52:29.301291 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 13:52:29.301846 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":288,"threads_started":1}
I20260927 13:52:29.304457 1399 heartbeater.cc:347] Connected to a master server at 127.20.133.126:41653
I20260927 13:52:29.304745 1399 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:29.305403 1399 heartbeater.cc:511] Master 127.20.133.126:41653 requested a full tablet report, sending...
I20260927 13:52:29.306965 907 ts_manager.cc:194] Registered new tserver with Master: 174e5ef292354ccc985b1b240e78ba26 (127.20.133.67:39645)
I20260927 13:52:29.307879 907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:33391
I20260927 13:52:29.319766 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:52:29.321657 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 989
Sep 27 13:52:29 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:52:29.341533 1404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:29.322752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.20.133.65:43859)
0927 13:52:29.323357 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:29.323364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:29.323391 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:52:29.323457 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:29.323464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:29.323695 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:29.323823 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:29.325485 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:29.326670 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:29.328383 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:29.328996 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:29.329115 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:29.334224 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:29.334329 (+ 105us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:29.334356 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:29.334446 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:29.338129 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:29.339091 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:29.339101 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:29.339103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:29.339299 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:29.339749 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:29.339752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:29.339755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:29.340146 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:29.340471 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:29.340866 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 13:52:29.341113 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":274,"threads_started":1}
I20260927 13:52:29.343384 1117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:29.428105 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 989
I20260927 13:52:29.442098 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 1128
Sep 27 13:52:29 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:52:29.467243 1406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:29.443457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.20.133.66:39053)
0927 13:52:29.444205 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:29.444213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:29.444245 (+ 32us) server_negotiation.cc:407] Connection header received
0927 13:52:29.444316 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:29.444322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:29.444649 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:29.444808 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:29.446269 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:29.447857 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:29.449620 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:29.450252 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:29.450401 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:29.460659 (+ 10258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:29.460680 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:29.460697 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:29.460744 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:29.464087 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:29.465159 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:29.465168 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:29.465170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:29.465356 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:29.465756 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:29.465759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:29.465762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:29.466124 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:29.466364 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:29.466710 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 13:52:29.466932 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":392,"threads_started":1}
I20260927 13:52:29.469012 1260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:29.554438 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 1128
I20260927 13:52:29.567919 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 1271
Sep 27 13:52:29 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:52:29.592540 1408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:29.569021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.20.133.67:39645)
0927 13:52:29.569718 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:29.569725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:29.569755 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:52:29.569814 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:29.569821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:29.570045 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:29.570213 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:29.571744 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:29.573023 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:29.574385 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:29.574968 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:29.575100 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:29.585465 (+ 10365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:29.585489 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:29.585511 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:29.585584 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:29.588887 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:29.589936 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:29.589944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:29.589947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:29.590104 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:29.590513 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:29.590516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:29.590518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:29.591020 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:29.591301 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:29.591779 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 13:52:29.592041 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":339,"threads_started":1}
I20260927 13:52:29.594089 1394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:29.678728 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 1271
I20260927 13:52:29.692976 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 876
I20260927 13:52:29.694685 938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:29.816488 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 876
I20260927 13:52:29.838320 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41653
--webserver_interface=127.20.133.126
--webserver_port=35765
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:41653
--ranger_config_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:42219/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.133.84:42219/kms with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:52:30.222380 1411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:30.222810 1411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:30.222955 1411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:30.231113 1411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:52:30.231307 1411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:30.231387 1411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:30.231472 1411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:52:30.231540 1411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:52:30.245230 1411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.133.84:42219/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:41653
--ranger_config_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=35765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:30.247368 1411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:30.249599 1411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:30.263566 1416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:30.263509 1417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:30.263671 1419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:30.264861 1411 server_base.cc:1034] running on GCE node
I20260927 13:52:30.265857 1411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:30.267676 1411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:30.268891 1411 hybrid_clock.cc:648] HybridClock initialized: now 1790517150268837 us; error 55 us; skew 500 ppm
I20260927 13:52:30.269347 1411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:30 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517150, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:30.273195 1411 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:52:30.275058 1411 webserver.cc:460] Webserver started at http://127.20.133.126:35765/ using document root <none> and password file <none>
I20260927 13:52:30.275888 1411 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:30.276029 1411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:52:30 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517150, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:52:30.328426 1411 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.005s
I20260927 13:52:30.333534 1427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:30.335772 1411 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260927 13:52:30.336067 1411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "cd60d9e46b214814945033fcb3821ff9"
format_stamp: "Formatted at 2026-09-27 13:52:25 on dist-test-slave-q25r"
server_key: "809bf5ac0deb772ebd02eac8c704c9d8"
server_key_iv: "33809b55388746a3a00032cc848272b7"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:30.337738 1411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:30.384809 1411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:30.389979 1411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:30.390301 1411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:30.412312 1411 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:41653
I20260927 13:52:30.412403 1478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:41653 every 8 connection(s)
I20260927 13:52:30.414201 1411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:52:30.416853 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 1411
I20260927 13:52:30.417804 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:42219/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:43859
--local_ip_for_outbound_sockets=127.20.133.65
--tserver_master_addrs=127.20.133.126:41653
--webserver_port=34205
--webserver_interface=127.20.133.65
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.133.84:42219/kms with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:52:30.422619 1479 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:52:30.426585 1479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9: Bootstrap starting.
I20260927 13:52:30.453730 1479 log.cc:824] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:30.467505 1479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:52:30.468266 1479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9: Bootstrap complete.
I20260927 13:52:30.482353 1479 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } }
I20260927 13:52:30.483835 1479 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd60d9e46b214814945033fcb3821ff9, State: Initialized, Role: FOLLOWER
I20260927 13:52:30.484886 1479 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } }
I20260927 13:52:30.485209 1479 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:52:30.485343 1479 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:52:30.485541 1479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:52:30.491876 1479 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } }
I20260927 13:52:30.492756 1479 leader_election.cc:302] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd60d9e46b214814945033fcb3821ff9; no voters:
I20260927 13:52:30.493669 1479 leader_election.cc:288] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:52:30.493921 1483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:52:30.495113 1483 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [term 2 LEADER]: Becoming Leader. State: Replica: cd60d9e46b214814945033fcb3821ff9, State: Running, Role: LEADER
I20260927 13:52:30.495822 1483 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } }
I20260927 13:52:30.496202 1479 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:52:30.498742 1484 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cd60d9e46b214814945033fcb3821ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } } }
I20260927 13:52:30.499174 1484 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:30.500020 1485 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd60d9e46b214814945033fcb3821ff9. Latest consensus state: current_term: 2 leader_uuid: "cd60d9e46b214814945033fcb3821ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd60d9e46b214814945033fcb3821ff9" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41653 } } }
I20260927 13:52:30.500510 1485 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9 [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:30.505336 1488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:52:30.512683 1488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:52:30.514473 1488 catalog_manager.cc:1259] Loaded cluster ID: 545f9d6bf5b24ee18d99f7a54ba4fc80
I20260927 13:52:30.514756 1488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:52:30.519544 1488 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:52:30.521436 1488 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9: Loaded TSK: 0
I20260927 13:52:30.522713 1479 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 13:52:30.523922 1488 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 13:52:30.947472 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:30.947939 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:30.948107 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:30.959890 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:30.960067 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:30.960273 1481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:52:30.976238 1481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.133.84:42219/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:43859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=34205
--tserver_master_addrs=127.20.133.126:41653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:30.978175 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:30.980571 1481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:31.021291 1517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:31.023121 1516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:31.027824 1519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:31.032647 1481 server_base.cc:1034] running on GCE node
I20260927 13:52:31.033771 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:31.035733 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:31.037017 1481 hybrid_clock.cc:648] HybridClock initialized: now 1790517151036950 us; error 59 us; skew 500 ppm
I20260927 13:52:31.037442 1481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:31 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517151, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:31.041986 1481 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:52:31.057687 1481 webserver.cc:460] Webserver started at http://127.20.133.65:34205/ using document root <none> and password file <none>
I20260927 13:52:31.058574 1481 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:31.058733 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:52:31 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517151, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:52:31.094774 1481 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 13:52:31.108781 1527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:31.111826 1481 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.004s sys 0.008s
I20260927 13:52:31.112125 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "f4815101859f427c80ccf7bc5f6618ff"
format_stamp: "Formatted at 2026-09-27 13:52:27 on dist-test-slave-q25r"
server_key: "ab1f6f77229b5ae4877062a379b0f9e2"
server_key_iv: "e6f47a3fb883a748e92e630feb497cf9"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:31.114012 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:31.136255 1481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:31.142999 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:31.143451 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:31.146322 1481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:31.148701 1481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:31.148838 1481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:31.148989 1481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:31.149084 1481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:31.204807 1481 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:43859
I20260927 13:52:31.204905 1640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:43859 every 8 connection(s)
I20260927 13:52:31.207311 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:52:31.213251 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 1481
I20260927 13:52:31.214268 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:42219/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:39053
--local_ip_for_outbound_sockets=127.20.133.66
--tserver_master_addrs=127.20.133.126:41653
--webserver_port=43811
--webserver_interface=127.20.133.66
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.133.84:42219/kms with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:52:31 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517151, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:31.265750 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:31.212000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:42523 (local address 127.20.133.126:41653)
0927 13:52:31.224257 (+ 12257us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:31.224285 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:31.233982 (+ 9697us) server_negotiation.cc:407] Connection header received
0927 13:52:31.237981 (+ 3999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:31.238011 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:31.238622 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:31.239084 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:31.242987 (+ 3903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:31.245192 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:31.248514 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:31.249658 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:31.249841 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:31.254805 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:31.254818 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:31.254840 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:31.254906 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:31.259137 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:31.260267 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:31.260279 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:31.260287 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:31.260564 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:31.261240 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:31.261246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:31.261250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:31.262077 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:31.262503 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:31.263777 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0927 13:52:31.264286 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11871,"thread_start_us":12251,"threads_started":1}
I20260927 13:52:31.267087 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:31.212395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:41653 (local address 127.20.133.65:42523)
0927 13:52:31.233640 (+ 21245us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:31.233749 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:31.236700 (+ 2951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:31.240908 (+ 4208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:31.240922 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:31.241670 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:31.242613 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:31.242646 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:31.245746 (+ 3100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:31.245754 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:31.248198 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:31.248221 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:31.250362 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:31.250370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:31.250679 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:31.251519 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:31.251556 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:31.254414 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:31.259365 (+ 4951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:31.259376 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:31.259394 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:31.260061 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:31.260765 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:31.260771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:31.260775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:31.261077 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:31.262657 (+ 1580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:31.262665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:31.263105 (+ 440us) client_negotiation.cc:769] Sending connection context
0927 13:52:31.264492 (+ 1387us) client_negotiation.cc:240] Negotiation successful
0927 13:52:31.265363 (+ 871us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20293,"thread_start_us":20010,"threads_started":1}
I20260927 13:52:31.271906 1641 heartbeater.cc:347] Connected to a master server at 127.20.133.126:41653
I20260927 13:52:31.272301 1641 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:31.273320 1641 heartbeater.cc:511] Master 127.20.133.126:41653 requested a full tablet report, sending...
W20260927 13:52:31.277702 1641 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:41653 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 13:52:31.278019 1641 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:31.278358 1641 heartbeater.cc:511] Master 127.20.133.126:41653 requested a full tablet report, sending...
I20260927 13:52:31.280373 1641 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:31.280862 1641 heartbeater.cc:511] Master 127.20.133.126:41653 requested a full tablet report, sending...
W20260927 13:52:31.282568 1641 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 13:52:31.632066 1645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:31.632408 1645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:31.632542 1645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:31.640997 1645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:31.641129 1645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:31.641279 1645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:52:31.657469 1645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.133.84:42219/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:39053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=43811
--tserver_master_addrs=127.20.133.126:41653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:31.659489 1645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:31.661319 1645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:31.677520 1660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:31.678040 1662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:31.678782 1659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:31.679849 1645 server_base.cc:1034] running on GCE node
I20260927 13:52:31.680670 1645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:31.681972 1645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:31.683254 1645 hybrid_clock.cc:648] HybridClock initialized: now 1790517151683178 us; error 61 us; skew 500 ppm
I20260927 13:52:31.683651 1645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:31 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517151, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:31.688149 1645 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:52:31.690330 1645 webserver.cc:460] Webserver started at http://127.20.133.66:43811/ using document root <none> and password file <none>
I20260927 13:52:31.691421 1645 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:31.691635 1645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:52:31 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517151, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:52:31.740199 1645 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.003s sys 0.016s
I20260927 13:52:31.744133 1672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:31.746419 1645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:52:31.746688 1645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "a423e7d653fe43eab1a8e28306c6a4f0"
format_stamp: "Formatted at 2026-09-27 13:52:28 on dist-test-slave-q25r"
server_key: "4e43288050e58e94c47d4b11381de0a9"
server_key_iv: "ea43e51a002b3d10da0f49ce0968d1d7"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:31.748144 1645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:31.791709 1645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:31.797894 1645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:31.798213 1645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:31.799510 1645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:31.801513 1645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:31.801646 1645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:31.801765 1645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:31.801842 1645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:52:31 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790517151, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:31.879459 1645 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:39053
I20260927 13:52:31.879552 1784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:39053 every 8 connection(s)
I20260927 13:52:31.881531 1645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:52:31.891837 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 1645
I20260927 13:52:31.892648 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.133.84:42219/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:39645
--local_ip_for_outbound_sockets=127.20.133.67
--tserver_master_addrs=127.20.133.126:41653
--webserver_port=39387
--webserver_interface=127.20.133.67
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.133.84:42219/kms with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:52:31 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517151, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:31.923046 1787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:31.884422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:48755 (local address 127.20.133.126:41653)
0927 13:52:31.904819 (+ 20397us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:31.904826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:31.904852 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:52:31.904929 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:31.904934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:31.905031 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:31.905244 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:31.907397 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:31.908512 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:31.911002 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:31.911596 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:31.911738 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:31.916483 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:31.916491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:31.916495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:31.916554 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:31.919569 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:31.920457 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:31.920462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:31.920464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:31.920601 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:31.921081 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:31.921085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:31.921087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:31.921414 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:31.921555 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:31.922431 (+ 876us) server_negotiation.cc:299] Negotiation successful
0927 13:52:31.922694 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20259,"thread_start_us":20108,"threads_started":1}
I20260927 13:52:31.924079 1788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:31.885087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:41653 (local address 127.20.133.66:48755)
0927 13:52:31.891180 (+ 6093us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:31.891253 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:31.894194 (+ 2941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:31.905582 (+ 11388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:31.905596 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:31.906266 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:31.907153 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:31.907181 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:31.908688 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:31.908694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:31.910775 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:31.910797 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:31.912099 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:31.912107 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:31.912436 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:31.913354 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:31.913384 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:31.916126 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:31.919753 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:31.919764 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:31.919782 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:31.920260 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:31.920718 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:31.920721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:31.920724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:31.920945 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:31.921635 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:31.921641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:31.921984 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 13:52:31.922242 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 13:52:31.922826 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5670,"thread_start_us":5355,"threads_started":1}
I20260927 13:52:31.926011 1785 heartbeater.cc:347] Connected to a master server at 127.20.133.126:41653
I20260927 13:52:31.926354 1785 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:31.927157 1785 heartbeater.cc:511] Master 127.20.133.126:41653 requested a full tablet report, sending...
I20260927 13:52:31.929237 1444 ts_manager.cc:194] Registered new tserver with Master: a423e7d653fe43eab1a8e28306c6a4f0 (127.20.133.66:39053)
I20260927 13:52:31.931526 1444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:48755
I20260927 13:52:32.282709 1641 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:32.283227 1641 heartbeater.cc:511] Master 127.20.133.126:41653 requested a full tablet report, sending...
I20260927 13:52:32.284488 1444 ts_manager.cc:194] Registered new tserver with Master: f4815101859f427c80ccf7bc5f6618ff (127.20.133.65:43859)
I20260927 13:52:32.285636 1444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:42523
Sep 27 13:52:32 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517151, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
W20260927 13:52:32.404311 1789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:32.404844 1789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:32.405033 1789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:32.416903 1789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:32.417291 1789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:32.417544 1789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:52:32.429878 1789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.133.84:42219/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:39645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=39387
--tserver_master_addrs=127.20.133.126:41653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:32.431222 1789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:32.432539 1789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:32.447346 1795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:32.449455 1796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:32.454202 1798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:32.454619 1789 server_base.cc:1034] running on GCE node
I20260927 13:52:32.455307 1789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:32.456483 1789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:32.457716 1789 hybrid_clock.cc:648] HybridClock initialized: now 1790517152457647 us; error 69 us; skew 500 ppm
I20260927 13:52:32.458056 1789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:32 dist-test-slave-q25r krb5kdc[432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517152, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:32.461665 1789 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:52:32.463749 1789 webserver.cc:460] Webserver started at http://127.20.133.67:39387/ using document root <none> and password file <none>
I20260927 13:52:32.464839 1789 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:32.465029 1789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:52:32 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517152, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
I20260927 13:52:32.494215 1789 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 13:52:32.498198 1806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:32.500257 1789 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:52:32.500516 1789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "174e5ef292354ccc985b1b240e78ba26"
format_stamp: "Formatted at 2026-09-27 13:52:29 on dist-test-slave-q25r"
server_key: "52abf76e02a1fd2d892383e28c860029"
server_key_iv: "326834b0edfc9f205bf36937616c773a"
server_key_version: "kuduclusterkey@0"
I20260927 13:52:32.501969 1789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:32.524628 1789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:32.530468 1789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:32.530827 1789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:32.532161 1789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:32.534147 1789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:32.534273 1789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:32.534387 1789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:32.534469 1789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:32.611284 1918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:39645 every 8 connection(s)
I20260927 13:52:32.613539 1789 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:39645
I20260927 13:52:32.615509 1789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:52:32.621368 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 1789
Sep 27 13:52:32 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517151, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for HTTP/127.20.133.84@KRBTEST.COM
Sep 27 13:52:32 dist-test-slave-q25r krb5kdc[432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517152, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:32.678053 1923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:32.621751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:41653 (local address 127.20.133.67:55155)
0927 13:52:32.640971 (+ 19220us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:32.641151 (+ 180us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:32.647121 (+ 5970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:32.648899 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:32.648913 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:32.651677 (+ 2764us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:32.652690 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:32.652730 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.654883 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.654890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:32.658491 (+ 3601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:32.658528 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.661029 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.661039 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:32.661382 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:32.665057 (+ 3675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:32.665096 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:32.667776 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:32.671873 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:32.671885 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:32.671905 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:32.672629 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:32.673680 (+ 1051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:32.673685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:32.673689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:32.674028 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:32.674861 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:32.674871 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:32.675385 (+ 514us) client_negotiation.cc:769] Sending connection context
0927 13:52:32.675741 (+ 356us) client_negotiation.cc:240] Negotiation successful
0927 13:52:32.676417 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18883,"thread_start_us":18659,"threads_started":1}
I20260927 13:52:32.679476 1925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:32.620747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:55155 (local address 127.20.133.126:41653)
0927 13:52:32.636766 (+ 16019us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:32.636774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:32.643661 (+ 6887us) server_negotiation.cc:407] Connection header received
0927 13:52:32.648107 (+ 4446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:32.648114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:32.648204 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:32.648308 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:32.652985 (+ 4677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.654301 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.659743 (+ 5442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.660453 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.660622 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:32.668114 (+ 7492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:32.668121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:32.668126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:32.668187 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:32.671610 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:32.673326 (+ 1716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:32.673332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:32.673335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:32.673523 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:32.674189 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:32.674194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:32.674198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:32.674608 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:32.674779 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:32.678679 (+ 3900us) server_negotiation.cc:299] Negotiation successful
0927 13:52:32.679021 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15861,"thread_start_us":23349,"threads_started":1}
I20260927 13:52:32.687564 1920 heartbeater.cc:347] Connected to a master server at 127.20.133.126:41653
I20260927 13:52:32.687969 1920 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:32.688745 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:32.622981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.20.133.126:41653)
0927 13:52:32.644751 (+ 21770us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:32.644756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:32.644776 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:52:32.644834 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:32.644840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:32.644923 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:32.645029 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:32.648755 (+ 3726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.652131 (+ 3376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.661077 (+ 8946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.661868 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.662030 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:32.666116 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:32.666125 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:32.666130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:32.666181 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:32.681351 (+ 15170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:32.682683 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:32.682690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:32.682693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:32.682932 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:32.683523 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:32.683528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:32.683532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:32.683929 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:32.684213 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:32.687961 (+ 3748us) server_negotiation.cc:299] Negotiation successful
0927 13:52:32.688252 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21621,"thread_start_us":13452,"threads_started":1}
I20260927 13:52:32.689029 1920 heartbeater.cc:511] Master 127.20.133.126:41653 requested a full tablet report, sending...
I20260927 13:52:32.690940 1443 ts_manager.cc:194] Registered new tserver with Master: 174e5ef292354ccc985b1b240e78ba26 (127.20.133.67:39645)
I20260927 13:52:32.692235 1443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:55155
I20260927 13:52:32.709960 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:52:32.761073 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:32.741714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.20.133.126:41653)
0927 13:52:32.741906 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:32.741913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:32.742635 (+ 722us) server_negotiation.cc:407] Connection header received
0927 13:52:32.742848 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:32.742855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:32.742944 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:32.743065 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:32.745783 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.746919 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.749062 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.749950 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.750099 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:32.753858 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:32.753866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:32.753871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:32.753921 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:32.757649 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:32.758700 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:32.758706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:32.758710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:32.758878 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:32.759423 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:32.759429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:32.759432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:32.759816 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:32.760064 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:32.760486 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 13:52:32.760744 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:52:32.768074 1443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:52:32.780277 1443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:52:32.845929 1937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:32.812095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:39645 (local address 127.0.0.1:50142)
0927 13:52:32.833768 (+ 21673us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:32.833823 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:32.834073 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:32.835230 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:32.835238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:32.835270 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:32.837197 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:32.837212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.839449 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.839455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:32.842253 (+ 2798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:32.842270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.844826 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.844834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:32.845071 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:32.845128 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 13:52:32.845262 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 13:52:32.845513 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21371,"thread_start_us":29993,"threads_started":1}
I20260927 13:52:32.848289 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:32.815165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.20.133.66:39053)
0927 13:52:32.834550 (+ 19385us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:32.834568 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:32.834892 (+ 324us) server_negotiation.cc:407] Connection header received
0927 13:52:32.835006 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:32.835025 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:32.837519 (+ 2494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:32.838333 (+ 814us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:32.838579 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.840075 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.844627 (+ 4552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.846238 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.847449 (+ 1211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:32.847683 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:32.847760 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 13:52:32.847945 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19218,"thread_start_us":20311,"threads_started":1}
I20260927 13:52:32.849035 1936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:32.812267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.20.133.67:39645)
0927 13:52:32.833255 (+ 20988us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:32.833266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:32.833972 (+ 706us) server_negotiation.cc:407] Connection header received
0927 13:52:32.834710 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:32.834719 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:32.835002 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:32.835226 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:32.837405 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.839235 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.842493 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.844201 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.844376 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:32.844663 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:32.848506 (+ 3843us) server_negotiation.cc:299] Negotiation successful
0927 13:52:32.848708 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20757,"thread_start_us":20566,"threads_started":1}
I20260927 13:52:32.847299 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:32.814908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:39053 (local address 127.0.0.1:47596)
0927 13:52:32.834615 (+ 19707us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:32.834634 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:32.834759 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:32.837742 (+ 2983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:32.837748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:32.837775 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:32.838112 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:32.838125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.840607 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.840613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:32.844389 (+ 3776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:32.844406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.846502 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.846510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:32.846694 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:32.846733 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 13:52:32.846801 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 13:52:32.846945 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19580,"thread_start_us":18401,"threads_started":1}
I20260927 13:52:32.857393 1720 tablet_service.cc:1467] Processing CreateTablet for tablet 670094215b0647efa0e742d4ac8b83ce (DEFAULT_TABLE table=test-table [id=db445777c459482f801cb55f6224c200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:52:32.858484 1854 tablet_service.cc:1467] Processing CreateTablet for tablet 670094215b0647efa0e742d4ac8b83ce (DEFAULT_TABLE table=test-table [id=db445777c459482f801cb55f6224c200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:52:32.860100 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:32.840685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:43859 (local address 127.0.0.1:39072)
0927 13:52:32.848675 (+ 7990us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:32.848697 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:32.848831 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:32.851334 (+ 2503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:32.851340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:32.851370 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:32.851696 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:32.851709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.854336 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.854343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:32.856603 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:32.856628 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.859109 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.859117 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:32.859325 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:32.859367 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 13:52:32.859498 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 13:52:32.859668 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7841,"thread_start_us":8206,"threads_started":1}
I20260927 13:52:32.860320 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670094215b0647efa0e742d4ac8b83ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:32.861734 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:32.834699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.20.133.65:43859)
0927 13:52:32.850460 (+ 15761us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:32.850469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:32.850498 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:52:32.850599 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:32.850607 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:32.850871 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:32.851076 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:32.852214 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.854053 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.856884 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:32.858433 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:32.858627 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:32.858892 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:32.860878 (+ 1986us) server_negotiation.cc:299] Negotiation successful
0927 13:52:32.861313 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15564,"thread_start_us":17115,"threads_started":1}
I20260927 13:52:32.866967 1576 tablet_service.cc:1467] Processing CreateTablet for tablet 670094215b0647efa0e742d4ac8b83ce (DEFAULT_TABLE table=test-table [id=db445777c459482f801cb55f6224c200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:52:32.869325 1576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670094215b0647efa0e742d4ac8b83ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:32.869326 1854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670094215b0647efa0e742d4ac8b83ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:32.894214 1941 tablet_bootstrap.cc:492] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: Bootstrap starting.
I20260927 13:52:32.900722 1941 tablet_bootstrap.cc:654] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:32.902626 1941 log.cc:824] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:32.912510 1945 tablet_bootstrap.cc:492] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: Bootstrap starting.
I20260927 13:52:32.913614 1941 tablet_bootstrap.cc:492] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: No bootstrap required, opened a new log
I20260927 13:52:32.914064 1941 ts_tablet_manager.cc:1397] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.014s
I20260927 13:52:32.914160 1944 tablet_bootstrap.cc:492] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: Bootstrap starting.
I20260927 13:52:32.916301 1945 tablet_bootstrap.cc:654] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:32.917649 1945 log.cc:824] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:32.917865 1944 tablet_bootstrap.cc:654] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:32.920868 1944 log.cc:824] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:32.927587 1945 tablet_bootstrap.cc:492] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: No bootstrap required, opened a new log
I20260927 13:52:32.927930 1945 ts_tablet_manager.cc:1397] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.003s
I20260927 13:52:32.936825 1785 heartbeater.cc:503] Master 127.20.133.126:41653 was elected leader, sending a full tablet report...
I20260927 13:52:32.937003 1941 raft_consensus.cc:348] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e5ef292354ccc985b1b240e78ba26" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39645 } } peers { permanent_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39053 } } peers { permanent_uuid: "f4815101859f427c80ccf7bc5f6618ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43859 } }
I20260927 13:52:32.937567 1941 raft_consensus.cc:374] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:32.937664 1941 raft_consensus.cc:729] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4815101859f427c80ccf7bc5f6618ff, State: Initialized, Role: FOLLOWER
I20260927 13:52:32.938432 1941 consensus_queue.cc:260] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e5ef292354ccc985b1b240e78ba26" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39645 } } peers { permanent_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39053 } } peers { permanent_uuid: "f4815101859f427c80ccf7bc5f6618ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43859 } }
I20260927 13:52:32.943459 1945 raft_consensus.cc:348] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e5ef292354ccc985b1b240e78ba26" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39645 } } peers { permanent_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39053 } } peers { permanent_uuid: "f4815101859f427c80ccf7bc5f6618ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43859 } }
I20260927 13:52:32.944018 1945 raft_consensus.cc:374] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:32.944151 1945 raft_consensus.cc:729] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a423e7d653fe43eab1a8e28306c6a4f0, State: Initialized, Role: FOLLOWER
I20260927 13:52:32.944562 1944 tablet_bootstrap.cc:492] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: No bootstrap required, opened a new log
I20260927 13:52:32.945007 1944 ts_tablet_manager.cc:1397] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.011s
I20260927 13:52:32.944971 1945 consensus_queue.cc:260] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e5ef292354ccc985b1b240e78ba26" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39645 } } peers { permanent_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39053 } } peers { permanent_uuid: "f4815101859f427c80ccf7bc5f6618ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43859 } }
I20260927 13:52:32.947698 1941 ts_tablet_manager.cc:1428] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: Time spent starting tablet: real 0.033s user 0.010s sys 0.009s
I20260927 13:52:32.948087 1641 heartbeater.cc:503] Master 127.20.133.126:41653 was elected leader, sending a full tablet report...
I20260927 13:52:32.953341 1945 ts_tablet_manager.cc:1428] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: Time spent starting tablet: real 0.025s user 0.011s sys 0.007s
I20260927 13:52:32.956612 1944 raft_consensus.cc:348] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e5ef292354ccc985b1b240e78ba26" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39645 } } peers { permanent_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39053 } } peers { permanent_uuid: "f4815101859f427c80ccf7bc5f6618ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43859 } }
I20260927 13:52:32.957176 1944 raft_consensus.cc:374] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:32.957299 1944 raft_consensus.cc:729] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174e5ef292354ccc985b1b240e78ba26, State: Initialized, Role: FOLLOWER
I20260927 13:52:32.957998 1944 consensus_queue.cc:260] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e5ef292354ccc985b1b240e78ba26" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39645 } } peers { permanent_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39053 } } peers { permanent_uuid: "f4815101859f427c80ccf7bc5f6618ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43859 } }
I20260927 13:52:32.959172 1920 heartbeater.cc:503] Master 127.20.133.126:41653 was elected leader, sending a full tablet report...
I20260927 13:52:32.959690 1944 ts_tablet_manager.cc:1428] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
W20260927 13:52:32.961946 1642 tablet.cc:2369] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:52:33.133803 1921 tablet.cc:2369] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:52:33.156091 1786 tablet.cc:2369] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:52:33.192227 1950 raft_consensus.cc:482] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:52:33.192593 1950 raft_consensus.cc:504] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e5ef292354ccc985b1b240e78ba26" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39645 } } peers { permanent_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39053 } } peers { permanent_uuid: "f4815101859f427c80ccf7bc5f6618ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43859 } }
I20260927 13:52:33.194303 1950 leader_election.cc:288] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 174e5ef292354ccc985b1b240e78ba26 (127.20.133.67:39645), a423e7d653fe43eab1a8e28306c6a4f0 (127.20.133.66:39053)
I20260927 13:52:33.202073 1936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:33.194710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:56857 (local address 127.20.133.67:39645)
0927 13:52:33.194907 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:33.194913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:33.195624 (+ 711us) server_negotiation.cc:407] Connection header received
0927 13:52:33.195887 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:33.195893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:33.195975 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:33.196087 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:33.196926 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.198034 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.200260 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.201103 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.201277 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:33.201351 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:33.201687 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 13:52:33.201815 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 13:52:33.202080 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:33.194736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:45615 (local address 127.20.133.66:39053)
0927 13:52:33.194921 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:33.194927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:33.195669 (+ 742us) server_negotiation.cc:407] Connection header received
0927 13:52:33.195887 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:33.195893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:33.195975 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:33.196098 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:33.196926 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.198049 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.200146 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.201063 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.201230 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:33.201323 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:33.201641 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 13:52:33.201787 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 13:52:33.202193 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:33.194699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:39053 (local address 127.20.133.65:45615)
0927 13:52:33.195570 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:33.195583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:33.195732 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:33.196165 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:33.196169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:33.196186 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:33.196727 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:33.196744 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.198224 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.198228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:33.199968 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:33.199982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.201304 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.201310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:33.201478 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:33.201505 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 13:52:33.201582 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 13:52:33.201716 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":55,"spinlock_wait_cycles":36608,"thread_start_us":458,"threads_started":1}
I20260927 13:52:33.202673 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:33.194642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:39645 (local address 127.20.133.65:56857)
0927 13:52:33.195501 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:33.195523 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:33.195696 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:33.196141 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:33.196147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:33.196177 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:33.196732 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:33.196743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.198206 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.198214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:33.200070 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:33.200089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.201359 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.201367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:33.201522 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:33.201547 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 13:52:33.201648 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 13:52:33.201786 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":98,"thread_start_us":472,"threads_started":1}
I20260927 13:52:33.203301 1740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670094215b0647efa0e742d4ac8b83ce" candidate_uuid: "f4815101859f427c80ccf7bc5f6618ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" is_pre_election: true
I20260927 13:52:33.203543 1874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670094215b0647efa0e742d4ac8b83ce" candidate_uuid: "f4815101859f427c80ccf7bc5f6618ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174e5ef292354ccc985b1b240e78ba26" is_pre_election: true
I20260927 13:52:33.203989 1874 raft_consensus.cc:2454] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4815101859f427c80ccf7bc5f6618ff in term 0.
I20260927 13:52:33.203974 1740 raft_consensus.cc:2454] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4815101859f427c80ccf7bc5f6618ff in term 0.
I20260927 13:52:33.204706 1529 leader_election.cc:302] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174e5ef292354ccc985b1b240e78ba26, f4815101859f427c80ccf7bc5f6618ff; no voters:
I20260927 13:52:33.205142 1950 raft_consensus.cc:2781] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:52:33.205304 1950 raft_consensus.cc:482] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:52:33.205408 1950 raft_consensus.cc:3037] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:33.207888 1950 raft_consensus.cc:504] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e5ef292354ccc985b1b240e78ba26" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39645 } } peers { permanent_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39053 } } peers { permanent_uuid: "f4815101859f427c80ccf7bc5f6618ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43859 } }
I20260927 13:52:33.208636 1950 leader_election.cc:288] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [CANDIDATE]: Term 1 election: Requested vote from peers 174e5ef292354ccc985b1b240e78ba26 (127.20.133.67:39645), a423e7d653fe43eab1a8e28306c6a4f0 (127.20.133.66:39053)
I20260927 13:52:33.209245 1740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670094215b0647efa0e742d4ac8b83ce" candidate_uuid: "f4815101859f427c80ccf7bc5f6618ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a423e7d653fe43eab1a8e28306c6a4f0"
I20260927 13:52:33.209245 1874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670094215b0647efa0e742d4ac8b83ce" candidate_uuid: "f4815101859f427c80ccf7bc5f6618ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174e5ef292354ccc985b1b240e78ba26"
I20260927 13:52:33.209475 1740 raft_consensus.cc:3037] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:33.209475 1874 raft_consensus.cc:3037] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:33.212651 1874 raft_consensus.cc:2454] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4815101859f427c80ccf7bc5f6618ff in term 1.
I20260927 13:52:33.212651 1740 raft_consensus.cc:2454] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4815101859f427c80ccf7bc5f6618ff in term 1.
I20260927 13:52:33.213186 1530 leader_election.cc:302] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a423e7d653fe43eab1a8e28306c6a4f0, f4815101859f427c80ccf7bc5f6618ff; no voters:
I20260927 13:52:33.213536 1950 raft_consensus.cc:2781] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:52:33.213958 1950 raft_consensus.cc:686] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 1 LEADER]: Becoming Leader. State: Replica: f4815101859f427c80ccf7bc5f6618ff, State: Running, Role: LEADER
I20260927 13:52:33.214790 1950 consensus_queue.cc:237] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e5ef292354ccc985b1b240e78ba26" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39645 } } peers { permanent_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39053 } } peers { permanent_uuid: "f4815101859f427c80ccf7bc5f6618ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43859 } }
I20260927 13:52:33.219635 1443 catalog_manager.cc:5495] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff reported cstate change: term changed from 0 to 1, leader changed from <none> to f4815101859f427c80ccf7bc5f6618ff (127.20.133.65). New cstate: current_term: 1 leader_uuid: "f4815101859f427c80ccf7bc5f6618ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e5ef292354ccc985b1b240e78ba26" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4815101859f427c80ccf7bc5f6618ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 43859 } health_report { overall_health: HEALTHY } } }
I20260927 13:52:33.279894 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:33.273933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.20.133.65:43859)
0927 13:52:33.274247 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:33.274254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:33.274462 (+ 208us) server_negotiation.cc:407] Connection header received
0927 13:52:33.274684 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:33.274688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:33.274779 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:33.274910 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:52:33.275370 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.276267 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.277777 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.278199 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.278299 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:33.278769 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:52:33.279299 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:52:33.279404 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:33.279559 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 13:52:33.279630 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260927 13:52:33.288677 1874 raft_consensus.cc:1261] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26 [term 1 FOLLOWER]: Refusing update from remote peer f4815101859f427c80ccf7bc5f6618ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:52:33.288677 1740 raft_consensus.cc:1261] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0 [term 1 FOLLOWER]: Refusing update from remote peer f4815101859f427c80ccf7bc5f6618ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:52:33.289808 1950 consensus_queue.cc:1035] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "174e5ef292354ccc985b1b240e78ba26" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:52:33.290117 1959 consensus_queue.cc:1035] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a423e7d653fe43eab1a8e28306c6a4f0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 39053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:52:33.303414 1964 mvcc.cc:205] Tried to move back new op lower bound from 7333958259859099648 to 7333958259572088832. Current Snapshot: MvccSnapshot[applied={T|T < 7333958259859099648}]
I20260927 13:52:33.318027 1444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:52:33.318293 1444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:52:33.322252 1444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd60d9e46b214814945033fcb3821ff9: Sending DeleteTablet for 3 replicas of tablet 670094215b0647efa0e742d4ac8b83ce
I20260927 13:52:33.323463 1720 tablet_service.cc:1514] Processing DeleteTablet for tablet 670094215b0647efa0e742d4ac8b83ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:52:33 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:47596
I20260927 13:52:33.323447 1854 tablet_service.cc:1514] Processing DeleteTablet for tablet 670094215b0647efa0e742d4ac8b83ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:52:33 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:50142
I20260927 13:52:33.324101 1576 tablet_service.cc:1514] Processing DeleteTablet for tablet 670094215b0647efa0e742d4ac8b83ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:52:33 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:39072
I20260927 13:52:33.324467 1972 tablet_replica.cc:330] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: stopping tablet replica
I20260927 13:52:33.325147 1972 raft_consensus.cc:2229] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:52:33.325718 1972 raft_consensus.cc:2258] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:52:33.327188 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 1481
I20260927 13:52:33.327289 1973 tablet_replica.cc:330] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: stopping tablet replica
I20260927 13:52:33.327745 1973 raft_consensus.cc:2229] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:52:33.328241 1973 raft_consensus.cc:2258] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:52:33.328539 1972 ts_tablet_manager.cc:1905] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:52:33.328846 1974 tablet_replica.cc:330] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: stopping tablet replica
I20260927 13:52:33.329295 1974 raft_consensus.cc:2229] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:52:33.330044 1974 raft_consensus.cc:2258] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:52:33.332367 1974 ts_tablet_manager.cc:1905] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:52:33.332366 1973 ts_tablet_manager.cc:1905] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:52:33.336149 1972 ts_tablet_manager.cc:1918] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:52:33.339692 1973 ts_tablet_manager.cc:1918] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:52:33.339998 1973 log.cc:1196] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/670094215b0647efa0e742d4ac8b83ce
I20260927 13:52:33.340689 1973 ts_tablet_manager.cc:1939] T 670094215b0647efa0e742d4ac8b83ce P 174e5ef292354ccc985b1b240e78ba26: Deleting consensus metadata
I20260927 13:52:33.341037 1974 ts_tablet_manager.cc:1918] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:52:33.341286 1974 log.cc:1196] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/670094215b0647efa0e742d4ac8b83ce
I20260927 13:52:33.341854 1974 ts_tablet_manager.cc:1939] T 670094215b0647efa0e742d4ac8b83ce P f4815101859f427c80ccf7bc5f6618ff: Deleting consensus metadata
I20260927 13:52:33.342959 1428 catalog_manager.cc:4841] TS 174e5ef292354ccc985b1b240e78ba26 (127.20.133.67:39645): tablet 670094215b0647efa0e742d4ac8b83ce (table test-table [id=db445777c459482f801cb55f6224c200]) successfully deleted
I20260927 13:52:33.343727 1429 catalog_manager.cc:4841] TS f4815101859f427c80ccf7bc5f6618ff (127.20.133.65:43859): tablet 670094215b0647efa0e742d4ac8b83ce (table test-table [id=db445777c459482f801cb55f6224c200]) successfully deleted
I20260927 13:52:33.344630 1972 log.cc:1196] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/670094215b0647efa0e742d4ac8b83ce
I20260927 13:52:33.345194 1972 ts_tablet_manager.cc:1939] T 670094215b0647efa0e742d4ac8b83ce P a423e7d653fe43eab1a8e28306c6a4f0: Deleting consensus metadata
I20260927 13:52:33.346742 1429 catalog_manager.cc:4841] TS a423e7d653fe43eab1a8e28306c6a4f0 (127.20.133.66:39053): tablet 670094215b0647efa0e742d4ac8b83ce (table test-table [id=db445777c459482f801cb55f6224c200]) successfully deleted
I20260927 13:52:33.355172 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:33.328047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.20.133.65:43859)
0927 13:52:33.328261 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:33.328268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:33.336896 (+ 8628us) server_negotiation.cc:407] Connection header received
0927 13:52:33.337198 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:33.337204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:33.337288 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:33.337363 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:33.339489 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.340798 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.342718 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.343393 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.343494 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:33.348588 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:33.348600 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:33.348616 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:33.348660 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:33.351936 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:33.352833 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:33.352840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:33.352843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:33.353014 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:33.353398 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:33.353402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:33.353404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:33.353813 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:33.354150 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:33.354547 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 13:52:33.354794 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:52:33.355966 1636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:33.459432 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 1481
I20260927 13:52:33.474218 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 1645
I20260927 13:52:33.488852 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:33.475094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.20.133.66:39053)
0927 13:52:33.475267 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:33.475272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:33.475316 (+ 44us) server_negotiation.cc:407] Connection header received
0927 13:52:33.475376 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:33.475380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:33.475467 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:33.475551 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:33.476982 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.477704 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.479060 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.479473 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.479599 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:33.482629 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:33.482646 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:33.482663 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:33.482702 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:33.485619 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:33.486361 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:33.486368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:33.486370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:33.486513 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:33.486890 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:33.486893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:33.486895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:33.487356 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:33.487648 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:33.487995 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 13:52:33.488278 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:52:33.489526 1780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:33.578773 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 1645
I20260927 13:52:33.592521 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 1789
I20260927 13:52:33.607846 1936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:33.593277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.20.133.67:39645)
0927 13:52:33.593445 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:33.593450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:33.593468 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:52:33.593558 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:33.593562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:33.593624 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:33.593706 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:33.595131 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.595917 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.597574 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:33.597995 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:33.598098 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:33.600963 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:33.600974 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:33.600997 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:33.601036 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:33.604221 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:33.605212 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:33.605221 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:33.605224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:33.605406 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:33.605818 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:33.605821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:33.605823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:33.606253 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:33.606493 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:33.606902 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 13:52:33.607531 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:52:33.608489 1914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:33.700398 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 1789
I20260927 13:52:33.715571 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 1411
I20260927 13:52:33.716730 1474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:33.833228 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 1411
I20260927 13:52:33.853204 21013 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 13:52:34.189453 21013 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 13:52:34.189657 21013 mini_ranger.cc:66] Stopping Ranger...
I20260927 13:52:34.525847 21013 mini_ranger.cc:68] Stopped Ranger
2026-09-27 13:52:34.526 UTC [462] LOG: received smart shutdown request
2026-09-27 13:52:34.530 UTC [462] LOG: background worker "logical replication launcher" (PID 471) exited with exit code 1
2026-09-27 13:52:34.530 UTC [466] LOG: shutting down
2026-09-27 13:52:34.700 UTC [462] LOG: database system is shut down
2026-09-27T13:52:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53842 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:52:34 dist-test-slave-q25r krb5kdc[1988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:52:34 dist-test-slave-q25r krb5kdc[1988](info): set up 2 sockets
Sep 27 13:52:34 dist-test-slave-q25r krb5kdc[1988](info): commencing operation
krb5kdc: starting...
W20260927 13:52:35.848634 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:52:36 dist-test-slave-q25r krb5kdc[1988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:52:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:52:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:52:36.112459 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:39337
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:36225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:39337,127.20.133.125:37985,127.20.133.124:46135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:52:36.510517 2001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:36.510934 2001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:36.511041 2001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:36.519217 2001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:52:36.519373 2001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:36.519456 2001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:36.519546 2001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:52:36.519608 2001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:52:36.530824 2001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:36225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:39337,127.20.133.125:37985,127.20.133.124:46135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:36.532608 2001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:36.534225 2001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:36.548007 2009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:36.548286 2006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:36.548828 2007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:36.548815 2001 server_base.cc:1034] running on GCE node
I20260927 13:52:36.549914 2001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:36.551505 2001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:36.552721 2001 hybrid_clock.cc:648] HybridClock initialized: now 1790517156552663 us; error 57 us; skew 500 ppm
I20260927 13:52:36.553133 2001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:36 dist-test-slave-q25r krb5kdc[1988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517156, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:36.556488 2001 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:52:36.558154 2001 webserver.cc:460] Webserver started at http://127.20.133.126:44233/ using document root <none> and password file <none>
I20260927 13:52:36.559006 2001 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:36.559137 2001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:36.559440 2001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:52:36.562912 2001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "f3c51301454b47f6b188dc465b6e22c5"
format_stamp: "Formatted at 2026-09-27 13:52:36 on dist-test-slave-q25r"
server_key: "916dec9d41ef3e9b5d8d7ae85d649f80"
server_key_iv: "1a4cfaf5afcf7fe19e786a304ce41647"
server_key_version: "encryptionkey@0"
I20260927 13:52:36.563808 2001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "f3c51301454b47f6b188dc465b6e22c5"
format_stamp: "Formatted at 2026-09-27 13:52:36 on dist-test-slave-q25r"
server_key: "916dec9d41ef3e9b5d8d7ae85d649f80"
server_key_iv: "1a4cfaf5afcf7fe19e786a304ce41647"
server_key_version: "encryptionkey@0"
I20260927 13:52:36.570004 2001 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 13:52:36.574385 2016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:36.576475 2001 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20260927 13:52:36.576717 2001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "f3c51301454b47f6b188dc465b6e22c5"
format_stamp: "Formatted at 2026-09-27 13:52:36 on dist-test-slave-q25r"
server_key: "916dec9d41ef3e9b5d8d7ae85d649f80"
server_key_iv: "1a4cfaf5afcf7fe19e786a304ce41647"
server_key_version: "encryptionkey@0"
I20260927 13:52:36.576936 2001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:36.593657 2001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:36.597957 2001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:36.598249 2001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:36.617789 2001 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:39337
I20260927 13:52:36.617790 2067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:39337 every 8 connection(s)
I20260927 13:52:36.619300 2001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:52:36.620216 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 2001
I20260927 13:52:36.620482 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:52:36.621085 21013 external_mini_cluster.cc:1428] Setting key bb47c6b76bc514b177a750c2774eb5aa
I20260927 13:52:36.623461 2068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:36.631426 2068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.133.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.125@KRBTEST.COM" created.
Sep 27 13:52:36 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517156, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:36.659946 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:36.635598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.20.133.126:39337)
0927 13:52:36.637916 (+ 2318us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:36.637922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:36.637970 (+ 48us) server_negotiation.cc:407] Connection header received
0927 13:52:36.640276 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:36.640287 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:36.640783 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:36.641059 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:36.643027 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:36.644484 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:36.646464 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:36.647081 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:36.647295 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:36.650989 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:36.650998 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:36.651012 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:36.651044 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:36.654488 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:36.655503 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:36.655516 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:36.655519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:36.655817 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:36.656496 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:36.656501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:36.656504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:36.657176 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:36.657361 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:36.658212 (+ 851us) server_negotiation.cc:299] Negotiation successful
0927 13:52:36.658808 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":295,"server-negotiator.queue_time_us":2121,"spinlock_wait_cycles":90368,"thread_start_us":1861,"threads_started":1}
I20260927 13:52:36.660007 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:36.635598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:39337 (local address 127.0.0.1:56102)
0927 13:52:36.636947 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:36.636999 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:36.639619 (+ 2620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:36.641187 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:36.641197 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:36.642097 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:36.642700 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:36.642735 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:36.644759 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:36.644852 (+ 93us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:36.646289 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:36.646307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:36.647345 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:36.647353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:36.647539 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:36.648235 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:36.648261 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:36.650747 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:36.654694 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:36.654706 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:36.654716 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:36.655200 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:36.656025 (+ 825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:36.656031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:36.656035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:36.656299 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:36.657374 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:36.657384 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:36.657693 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 13:52:36.657980 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 13:52:36.658634 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":1,"thread_start_us":712,"threads_started":1}
I20260927 13:52:36.663136 2068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:52:36.664374 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:36.663830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40602)
0927 13:52:36.664064 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:36.664200 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 13:52:36.664674 2018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
I20260927 13:52:36.666428 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:36.665937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40616)
0927 13:52:36.666136 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:36.666232 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 13:52:36.666656 2068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:37985: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab.
Entry for principal kudu/127.20.133.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab.
I20260927 13:52:36.676232 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.125:37985
--webserver_interface=127.20.133.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab
--principal=kudu/127.20.133.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:36225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:39337,127.20.133.125:37985,127.20.133.124:46135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:52:36.716046 2068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } attempt: 1
I20260927 13:52:36.717445 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:36.716880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40622)
0927 13:52:36.717117 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:36.717273 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 13:52:36.719233 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:36.718873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40624)
0927 13:52:36.719016 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:36.719094 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 13:52:36.719405 2068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:37985: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
I20260927 13:52:36.787716 2068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } attempt: 2
I20260927 13:52:36.789134 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:36.788635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40640)
0927 13:52:36.788886 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:36.788983 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 13:52:36.790134 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:36.789753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40648)
0927 13:52:36.789901 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:36.789997 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 13:52:36.790323 2068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:37985: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
I20260927 13:52:36.881608 2068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } attempt: 3
I20260927 13:52:36.882674 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:36.882243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40652)
0927 13:52:36.882433 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:36.882542 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 13:52:36.883594 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:36.883231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40658)
0927 13:52:36.883365 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:36.883435 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 13:52:36.883755 2068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:37985: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
I20260927 13:52:37.027019 2068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } attempt: 4
W20260927 13:52:37.027302 2075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:37.027626 2075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:37.027761 2075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 13:52:37.028322 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.027733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40668)
0927 13:52:37.028046 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.028151 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260927 13:52:37.029351 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.028969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40680)
0927 13:52:37.029105 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.029209 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 13:52:37.029589 2068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:37985: Network error: Client connection negotiation failed: client connection to 127.20.133.125:37985: connect: Connection refused (error 111)
W20260927 13:52:37.036118 2075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:52:37.036262 2075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:37.036317 2075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:37.036423 2075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:52:37.036478 2075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:52:37.046701 2075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:36225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:39337,127.20.133.125:37985,127.20.133.124:46135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab
--principal=kudu/127.20.133.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.125:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.133.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:37.048108 2075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:37.049417 2075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:37.060884 2080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:37.061026 2083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:37.061013 2081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:37.062083 2075 server_base.cc:1034] running on GCE node
I20260927 13:52:37.062877 2075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:37.064301 2075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:37.065594 2075 hybrid_clock.cc:648] HybridClock initialized: now 1790517157065525 us; error 46 us; skew 500 ppm
I20260927 13:52:37.065994 2075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517157, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:37.069342 2075 init.cc:377] Logged in from keytab as kudu/127.20.133.125@KRBTEST.COM (short username kudu)
I20260927 13:52:37.070966 2075 webserver.cc:460] Webserver started at http://127.20.133.125:33367/ using document root <none> and password file <none>
I20260927 13:52:37.071700 2075 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:37.071811 2075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:37.072085 2075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:52:37.075335 2075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/data/instance:
uuid: "3c3df47f4ae9402eb546ff4d77bc090d"
format_stamp: "Formatted at 2026-09-27 13:52:37 on dist-test-slave-q25r"
server_key: "1d5dcc33dff2c8e5280f45fa265f966f"
server_key_iv: "644a4b85c3837323c7584a1b2c89ac88"
server_key_version: "encryptionkey@0"
I20260927 13:52:37.076171 2075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/wal/instance:
uuid: "3c3df47f4ae9402eb546ff4d77bc090d"
format_stamp: "Formatted at 2026-09-27 13:52:37 on dist-test-slave-q25r"
server_key: "1d5dcc33dff2c8e5280f45fa265f966f"
server_key_iv: "644a4b85c3837323c7584a1b2c89ac88"
server_key_version: "encryptionkey@0"
I20260927 13:52:37.081681 2075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 13:52:37.085778 2090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:37.087406 2075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:52:37.087607 2075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/wal
uuid: "3c3df47f4ae9402eb546ff4d77bc090d"
format_stamp: "Formatted at 2026-09-27 13:52:37 on dist-test-slave-q25r"
server_key: "1d5dcc33dff2c8e5280f45fa265f966f"
server_key_iv: "644a4b85c3837323c7584a1b2c89ac88"
server_key_version: "encryptionkey@0"
I20260927 13:52:37.087829 2075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:37.110950 2075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:37.115432 2075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:37.115689 2075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:37.134020 2075 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.125:37985
I20260927 13:52:37.134049 2141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.125:37985 every 8 connection(s)
I20260927 13:52:37.135435 2075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
I20260927 13:52:37.139204 2142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:37.139213 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 2075
I20260927 13:52:37.139362 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-1/wal/instance
I20260927 13:52:37.139792 21013 external_mini_cluster.cc:1428] Setting key 3777e619f5d8e2cf02256fd00c75bc45
I20260927 13:52:37.145962 2142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.133.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.124@KRBTEST.COM" created.
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517157, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:37.170542 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.148657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.20.133.126:39337)
0927 13:52:37.148825 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.148830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.150328 (+ 1498us) server_negotiation.cc:407] Connection header received
0927 13:52:37.152255 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.152262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.152371 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.152464 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.154064 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.154743 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.156443 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.156857 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.156977 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.163233 (+ 6256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.163241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.163246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.163295 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.166612 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.167665 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.167672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.167676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.167833 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.168478 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.168483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.168486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.168841 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.168989 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.169875 (+ 886us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.170151 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:52:37.171583 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.149326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:39337 (local address 127.0.0.1:56118)
0927 13:52:37.150150 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.150195 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:37.151967 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:37.152731 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:37.152740 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:37.153296 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:37.153883 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.153901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.154903 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.154907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.156276 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.156291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.157106 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.157114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.157340 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.158001 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:37.158032 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:37.162884 (+ 4852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:37.166936 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.166946 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.166970 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.167533 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.168007 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.168011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.168014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.168308 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.169005 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:37.169012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:37.169403 (+ 391us) client_negotiation.cc:769] Sending connection context
0927 13:52:37.169714 (+ 311us) client_negotiation.cc:240] Negotiation successful
0927 13:52:37.170305 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":356,"threads_started":1}
I20260927 13:52:37.173537 2142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517157, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab.
Entry for principal kudu/127.20.133.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab.
I20260927 13:52:37.189599 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.124:46135
--webserver_interface=127.20.133.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab
--principal=kudu/127.20.133.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:36225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:39337,127.20.133.125:37985,127.20.133.124:46135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:52:37.189953 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.174157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40688)
0927 13:52:37.174375 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.174395 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:37.174520 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:37.175565 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:37.175570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:37.175995 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:37.176132 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.176140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.177459 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.177464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.178289 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.178301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.179108 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.179113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.179317 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.179856 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:37.179874 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:37.183830 (+ 3956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:37.187320 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.187325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.187330 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.187707 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.188085 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.188090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.188094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.188225 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.188964 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:37.188969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:37.189114 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 13:52:37.189233 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 13:52:37.189551 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 13:52:37.190021 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.174452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.20.133.125:37985)
0927 13:52:37.175149 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.175157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.175183 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:52:37.175226 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.175232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.175414 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.175546 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.176281 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.177332 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.178418 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.178974 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.179084 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.184058 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.184074 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.184090 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.184128 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.187170 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.187849 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.187856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.187858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.187981 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.188388 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.188393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.188395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.188827 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.189043 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.189389 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.189655 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":520,"thread_start_us":384,"threads_started":1}
I20260927 13:52:37.192637 2142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:52:37.193894 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.193367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54694)
0927 13:52:37.193583 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.193714 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 13:52:37.194173 2092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
I20260927 13:52:37.195823 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.195436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54698)
0927 13:52:37.195632 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.195713 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 13:52:37.196048 2142 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46135: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
I20260927 13:52:37.245388 2142 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } attempt: 1
I20260927 13:52:37.246524 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.246051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54704)
0927 13:52:37.246266 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.246370 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 13:52:37.247627 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.247267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54710)
0927 13:52:37.247428 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.247503 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 13:52:37.247824 2142 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46135: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
I20260927 13:52:37.316067 2142 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } attempt: 2
I20260927 13:52:37.317549 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.316906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54724)
0927 13:52:37.317205 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.317339 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 13:52:37.318634 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.318187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54728)
0927 13:52:37.318378 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.318475 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 13:52:37.318871 2142 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46135: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
I20260927 13:52:37.328850 2068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } attempt: 5
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517156, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
I20260927 13:52:37.343321 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.329864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.20.133.125:37985)
0927 13:52:37.330037 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.330041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.330063 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:52:37.330143 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.330148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.330213 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.330303 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.331264 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.331968 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.333671 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.334157 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.334269 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.338122 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.338127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.338131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.338174 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.340790 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.341571 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.341575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.341577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.341696 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.342067 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.342070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.342072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.342351 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.342461 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.342903 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.343076 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:52:37.343547 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.329647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40700)
0927 13:52:37.329864 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.329886 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:37.330024 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:37.330357 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:37.330362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:37.330834 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:37.331058 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.331070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.332121 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.332127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.333503 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.333519 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.334321 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.334327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.334521 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.335143 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:37.335166 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:37.337899 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:37.340945 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.340950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.340954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.341462 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.341804 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.341809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.341812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.341978 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.342461 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:37.342466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:37.342633 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 13:52:37.342789 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 13:52:37.343154 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 13:52:37.344648 2068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:52:37.345726 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.345302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54738)
0927 13:52:37.345486 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.345586 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 13:52:37.346624 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.346290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54748)
0927 13:52:37.346420 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.346524 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 13:52:37.346804 2068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46135: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
I20260927 13:52:37.398031 2068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } attempt: 1
I20260927 13:52:37.399276 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.398793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54760)
0927 13:52:37.399008 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.399116 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 13:52:37.400374 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.399997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54768)
0927 13:52:37.400143 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.400217 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 13:52:37.400580 2068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46135: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
I20260927 13:52:37.410095 2142 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } attempt: 3
I20260927 13:52:37.411412 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.410803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54772)
0927 13:52:37.411063 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.411186 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 13:52:37.412593 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.412086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54778)
0927 13:52:37.412321 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.412447 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260927 13:52:37.412842 2142 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46135: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
I20260927 13:52:37.468786 2068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } attempt: 2
I20260927 13:52:37.470028 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.469500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54786)
0927 13:52:37.469746 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.469841 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 13:52:37.471021 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.470625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54794)
0927 13:52:37.470811 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.470902 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 13:52:37.471232 2068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46135: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
W20260927 13:52:37.521075 2149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:37.521384 2149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:37.521516 2149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:37.528788 2149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:52:37.528915 2149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:37.528992 2149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:37.529064 2149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:52:37.529125 2149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:52:37.539059 2149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:36225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:39337,127.20.133.125:37985,127.20.133.124:46135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab
--principal=kudu/127.20.133.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.124:46135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.133.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:37.540473 2149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:37.541779 2149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:37.553303 2154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:37.553254 2157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:37.553269 2155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:37.554116 2149 server_base.cc:1034] running on GCE node
I20260927 13:52:37.554922 2149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:37.556052 2142 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } attempt: 4
I20260927 13:52:37.556489 2149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:37.557204 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.556684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54798)
0927 13:52:37.556919 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.557018 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 13:52:37.557691 2149 hybrid_clock.cc:648] HybridClock initialized: now 1790517157557635 us; error 53 us; skew 500 ppm
I20260927 13:52:37.558081 2149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:52:37.558209 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.557838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54808)
0927 13:52:37.557986 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.558080 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 13:52:37.558420 2142 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46135: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517157, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:37.561195 2149 init.cc:377] Logged in from keytab as kudu/127.20.133.124@KRBTEST.COM (short username kudu)
I20260927 13:52:37.562705 2149 webserver.cc:460] Webserver started at http://127.20.133.124:33437/ using document root <none> and password file <none>
I20260927 13:52:37.563395 2149 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:37.563506 2149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:37.563804 2149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:52:37.566811 2149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/data/instance:
uuid: "e055b2928fb4437e98cb0c809cd15172"
format_stamp: "Formatted at 2026-09-27 13:52:37 on dist-test-slave-q25r"
server_key: "3fc93f79a48f1dfc154fbce2da1d56fb"
server_key_iv: "8ff64cca820f6b87fca340b7ec553c0b"
server_key_version: "encryptionkey@0"
I20260927 13:52:37.567585 2149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/wal/instance:
uuid: "e055b2928fb4437e98cb0c809cd15172"
format_stamp: "Formatted at 2026-09-27 13:52:37 on dist-test-slave-q25r"
server_key: "3fc93f79a48f1dfc154fbce2da1d56fb"
server_key_iv: "8ff64cca820f6b87fca340b7ec553c0b"
server_key_version: "encryptionkey@0"
I20260927 13:52:37.572839 2149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 13:52:37.576489 2164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:37.577442 2068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } attempt: 3
I20260927 13:52:37.577894 2149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:52:37.578085 2149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/wal
uuid: "e055b2928fb4437e98cb0c809cd15172"
format_stamp: "Formatted at 2026-09-27 13:52:37 on dist-test-slave-q25r"
server_key: "3fc93f79a48f1dfc154fbce2da1d56fb"
server_key_iv: "8ff64cca820f6b87fca340b7ec553c0b"
server_key_version: "encryptionkey@0"
I20260927 13:52:37.578315 2149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:37.578611 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.578138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54820)
0927 13:52:37.578327 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.578420 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 13:52:37.579610 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.579281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54834)
0927 13:52:37.579411 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.579508 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 13:52:37.579802 2068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46135: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46135: connect: Connection refused (error 111)
I20260927 13:52:37.596596 2149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:37.600754 2149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:37.601004 2149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:37.618636 2149 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.124:46135
I20260927 13:52:37.618618 2215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.124:46135 every 8 connection(s)
I20260927 13:52:37.620186 2149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
I20260927 13:52:37.623032 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 2149
I20260927 13:52:37.623240 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516948056776-21013-0/minicluster-data/master-2/wal/instance
I20260927 13:52:37.623749 21013 external_mini_cluster.cc:1428] Setting key 15e315538ea537d63f6596c8f0377cd1
I20260927 13:52:37.624255 2216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:37.633246 2216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517157, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:37.648599 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.625576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.20.133.124:46135)
0927 13:52:37.626496 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.626508 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.627023 (+ 515us) server_negotiation.cc:407] Connection header received
0927 13:52:37.629171 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.629193 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.629602 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.629973 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.631433 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.632754 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.634156 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.634828 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.634988 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.640019 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.640028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.640039 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.640082 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.643830 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.644696 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.644703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.644712 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.644895 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.645339 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.645344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.645347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.646088 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.646376 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.646746 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.647165 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":481,"threads_started":1}
I20260927 13:52:37.651190 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.635696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:39337 (local address 127.0.0.1:56128)
0927 13:52:37.636520 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.636564 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:37.636740 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:37.637149 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:37.637156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:37.638048 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:37.638383 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.638393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.639514 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.639519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.640835 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.640851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.642025 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.642034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.642235 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.642798 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:37.642820 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:37.645061 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:37.648632 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.648641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.648646 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.649157 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.649564 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.649568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.649570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.649703 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.650185 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:37.650193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:37.650361 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 13:52:37.650511 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 13:52:37.650819 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":381,"threads_started":1}
I20260927 13:52:37.651607 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.635749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.20.133.126:39337)
0927 13:52:37.635962 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.635970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.636661 (+ 691us) server_negotiation.cc:407] Connection header received
0927 13:52:37.636896 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.636901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.636972 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.637061 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.638565 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.639351 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.641018 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.641775 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.641927 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.645273 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.645280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.645284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.645328 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.648500 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.649283 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.649287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.649289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.649456 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.649813 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.649817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.649819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.650062 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.650214 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.650657 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.651265 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:52:37.653023 2216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517157, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:37.666070 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.653660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:37985 (local address 127.0.0.1:40716)
0927 13:52:37.653823 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.653843 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:37.653937 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:37.654263 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:37.654266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:37.654569 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:37.654731 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.654739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.655771 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.655777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.657004 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.657017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.657864 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.657870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.658065 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.658595 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:37.658614 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:37.660690 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:37.663705 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.663711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.663716 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.664100 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.664502 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.664508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.664512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.664654 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.665135 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:37.665140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:37.665295 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 13:52:37.665414 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 13:52:37.665709 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 13:52:37.666096 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.653823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40716 (local address 127.20.133.125:37985)
0927 13:52:37.653970 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.653976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.653994 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:52:37.654052 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.654057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.654133 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.654248 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.654864 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.655629 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.657153 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.657709 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.657824 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.660922 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.660928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.660931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.660963 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.663567 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.664215 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.664221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.664223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.664400 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.664751 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.664757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.664760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.665026 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.665157 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.665515 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.665729 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 13:52:37.666728 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.651405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.20.133.126:39337)
0927 13:52:37.651993 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.651998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.652159 (+ 161us) server_negotiation.cc:407] Connection header received
0927 13:52:37.652299 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.652302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.652385 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.652468 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.654075 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.655056 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.656670 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.657144 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.657277 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.661473 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.661479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.661482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.661517 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.664077 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.664841 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.664846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.664849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.664969 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.665333 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.665337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.665339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.665609 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.665851 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.666155 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.666377 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":257,"threads_started":1}
I20260927 13:52:37.667531 2216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517157, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
I20260927 13:52:37.680536 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.668224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54848)
0927 13:52:37.668420 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.668437 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:37.668545 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:37.668850 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:37.668855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:37.669226 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:37.669356 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.669364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.670275 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.670280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.671100 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.671111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.671940 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.671946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.672113 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.672742 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:37.672759 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:37.674911 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:37.677850 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.677856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.677860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.678271 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.678691 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.678700 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.678704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.678899 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.679571 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:37.679575 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:37.679717 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 13:52:37.679888 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 13:52:37.680189 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 13:52:37.680825 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.668473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54848 (local address 127.20.133.124:46135)
0927 13:52:37.668615 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.668619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.668633 (+ 14us) server_negotiation.cc:407] Connection header received
0927 13:52:37.668669 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.668674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.668746 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.668836 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.669477 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.670159 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.671218 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.671785 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.671894 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.675129 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.675134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.675138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.675167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.677705 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.678380 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.678385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.678389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.678518 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.679030 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.679041 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.679044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.679444 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.679628 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.679994 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.680422 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":45}
I20260927 13:52:37.684885 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.668873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.20.133.125:37985)
0927 13:52:37.669018 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.669023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.669099 (+ 76us) server_negotiation.cc:407] Connection header received
0927 13:52:37.669253 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.669259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.669320 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.669393 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.670533 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.671244 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.672715 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.673175 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.673287 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.678074 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.678080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.678083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.678119 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.681550 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.682516 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.682520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.682523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.682675 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.683154 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.683160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.683163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.683530 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.683754 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.684306 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.684567 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:52:37.686872 2216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172: Bootstrap starting.
I20260927 13:52:37.690202 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:52:37.690728 2216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:37.692022 2216 log.cc:824] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:37.695766 2216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172: No bootstrap required, opened a new log
I20260927 13:52:37.707379 2216 raft_consensus.cc:348] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } }
I20260927 13:52:37.708065 2216 raft_consensus.cc:374] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:37.708244 2216 raft_consensus.cc:729] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e055b2928fb4437e98cb0c809cd15172, State: Initialized, Role: FOLLOWER
I20260927 13:52:37.709184 2216 consensus_queue.cc:260] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } }
I20260927 13:52:37.716370 2230 sys_catalog.cc:455] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } } }
I20260927 13:52:37.716949 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.697913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.20.133.125:37985)
0927 13:52:37.698100 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.698106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.698461 (+ 355us) server_negotiation.cc:407] Connection header received
0927 13:52:37.698787 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.698793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.698881 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.698948 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.701455 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.702594 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.705021 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.705682 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.705785 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.709705 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.709712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.709716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.709763 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.713377 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.714436 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.714442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.714447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.714633 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.715086 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.715092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.715096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.715506 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.715757 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.716329 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.716610 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 13:52:37.718050 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.696810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.20.133.126:39337)
0927 13:52:37.697028 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.697035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.697522 (+ 487us) server_negotiation.cc:407] Connection header received
0927 13:52:37.698460 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.698468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.698570 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.698652 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.701554 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.702745 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.705292 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.706046 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.706146 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.711053 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.711060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.711064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.711109 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.714652 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.715466 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.715472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.715475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.715634 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.716122 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.716128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.716132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.716529 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.716774 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.717464 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.717713 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:52:37.719049 2230 sys_catalog.cc:458] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:52:37.718907 2216 sys_catalog.cc:564] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:52:37.727150 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.698733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.20.133.124:46135)
0927 13:52:37.698928 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.698933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.708911 (+ 9978us) server_negotiation.cc:407] Connection header received
0927 13:52:37.709085 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.709091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.709182 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.709282 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.711152 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.712263 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.714391 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.715091 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.715192 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.720155 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.720167 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.720171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.720213 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.723671 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.724403 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.724407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.724410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.724599 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.724987 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.724991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.724993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.725287 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.726472 (+ 1185us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.726613 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.726788 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 13:52:37.730348 2241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:52:37.730473 2241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:52:37.727790 2149 master_runner.cc:186] Error getting master registration for 127.20.133.126:39337: OK, instance_id { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" instance_seqno: 1790517156612505 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:52:37.739889 2149 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 13:52:37.757021 2068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } attempt: 4
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517156, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
I20260927 13:52:37.772969 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.757646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54870)
0927 13:52:37.757851 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.757871 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:37.757968 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:37.758324 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:37.758328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:37.758708 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:37.758906 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.758917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.761074 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.761080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.762123 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.762137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.762914 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.762921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.763077 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.763673 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:37.763691 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:37.765940 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:37.770573 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.770577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.770580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.770970 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.771387 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.771390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.771392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.771531 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.772161 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:37.772164 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:37.772303 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 13:52:37.772457 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 13:52:37.772694 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 13:52:37.773386 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.757738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.20.133.124:46135)
0927 13:52:37.757958 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.757964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.757985 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:52:37.758051 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.758056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.758150 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.758273 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.759057 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.760667 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.762279 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.762735 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.762849 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.766147 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.766154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.766159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.766201 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.770422 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.771112 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.771118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.771121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.771286 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.771631 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.771640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.771643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.772034 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.772189 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.772580 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.772805 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 13:52:37.778685 2068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5: Bootstrap starting.
I20260927 13:52:37.782307 2068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:37.783372 2068 log.cc:824] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:37.786120 2068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5: No bootstrap required, opened a new log
I20260927 13:52:37.794198 2068 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } }
I20260927 13:52:37.794531 2068 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:37.794637 2068 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c51301454b47f6b188dc465b6e22c5, State: Initialized, Role: FOLLOWER
I20260927 13:52:37.795190 2068 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } }
I20260927 13:52:37.796077 2244 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } } }
I20260927 13:52:37.796490 2244 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:52:37.797287 2068 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 13:52:37.806421 2255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:52:37.806641 2255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:52:37.806000 2001 master_runner.cc:186] Error getting master registration for 127.20.133.125:37985: OK, instance_id { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" instance_seqno: 1790517157128532 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:52:37.813614 2001 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 13:52:37.852319 2022 tablet.cc:2369] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:52:37.853142 2170 tablet.cc:2369] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:52:37.857620 2142 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } attempt: 5
Sep 27 13:52:37 dist-test-slave-q25r krb5kdc[1988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517157, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
I20260927 13:52:37.873522 2230 raft_consensus.cc:482] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:52:37.873793 2230 raft_consensus.cc:504] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } }
I20260927 13:52:37.875316 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.858320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46135 (local address 127.0.0.1:54886)
0927 13:52:37.858616 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:37.858637 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:37.858759 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:37.859171 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:37.859176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:37.859717 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:37.859933 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.859947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.861514 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.861531 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.862967 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:37.862983 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.864064 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.864071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:37.864317 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.865034 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:37.865060 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:37.867901 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:37.872279 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.872285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.872290 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.872913 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.873413 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:37.873418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:37.873422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:37.873603 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:37.874272 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:37.874277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:37.874435 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 13:52:37.874606 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 13:52:37.874920 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260927 13:52:37.875362 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:37.858515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.20.133.124:46135)
0927 13:52:37.858712 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:37.858718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:37.858740 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:52:37.858912 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:37.858917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:37.858996 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:37.859124 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:37.860129 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.861313 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.863150 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:37.863889 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:37.864022 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:37.868192 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:37.868200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:37.868210 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:37.868253 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:37.872108 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.873081 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.873088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.873092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.873259 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:37.873733 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:37.873739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:37.873742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:37.874136 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:37.874314 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:37.874719 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 13:52:37.874974 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 13:52:37.875418 2230 leader_election.cc:288] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c51301454b47f6b188dc465b6e22c5 (127.20.133.126:39337), 3c3df47f4ae9402eb546ff4d77bc090d (127.20.133.125:37985)
I20260927 13:52:37.876464 2043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e055b2928fb4437e98cb0c809cd15172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c51301454b47f6b188dc465b6e22c5" is_pre_election: true
I20260927 13:52:37.877141 2043 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e055b2928fb4437e98cb0c809cd15172 in term 0.
I20260927 13:52:37.877807 2166 leader_election.cc:302] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e055b2928fb4437e98cb0c809cd15172, f3c51301454b47f6b188dc465b6e22c5; no voters:
I20260927 13:52:37.878403 2230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:52:37.878569 2230 raft_consensus.cc:482] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:52:37.878669 2230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:37.881719 2142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d: Bootstrap starting.
I20260927 13:52:37.882097 2230 raft_consensus.cc:504] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } }
I20260927 13:52:37.884992 2043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e055b2928fb4437e98cb0c809cd15172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c51301454b47f6b188dc465b6e22c5"
I20260927 13:52:37.885211 2043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:37.886237 2142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:37.887624 2142 log.cc:824] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:37.888665 2043 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e055b2928fb4437e98cb0c809cd15172 in term 1.
I20260927 13:52:37.889271 2166 leader_election.cc:302] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e055b2928fb4437e98cb0c809cd15172, f3c51301454b47f6b188dc465b6e22c5; no voters:
I20260927 13:52:37.891942 2230 leader_election.cc:288] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [CANDIDATE]: Term 1 election: Requested vote from peers f3c51301454b47f6b188dc465b6e22c5 (127.20.133.126:39337), 3c3df47f4ae9402eb546ff4d77bc090d (127.20.133.125:37985)
I20260927 13:52:37.892541 2230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:52:37.893244 2230 raft_consensus.cc:686] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [term 1 LEADER]: Becoming Leader. State: Replica: e055b2928fb4437e98cb0c809cd15172, State: Running, Role: LEADER
I20260927 13:52:37.894094 2230 consensus_queue.cc:237] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } }
I20260927 13:52:37.903352 2258 sys_catalog.cc:455] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e055b2928fb4437e98cb0c809cd15172. Latest consensus state: current_term: 1 leader_uuid: "e055b2928fb4437e98cb0c809cd15172" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } } }
I20260927 13:52:37.903821 2258 sys_catalog.cc:458] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:37.905843 2260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:52:37.906926 2142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d: No bootstrap required, opened a new log
I20260927 13:52:37.910918 2260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:52:37.921896 2142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } }
I20260927 13:52:37.922477 2142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:37.922631 2142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c3df47f4ae9402eb546ff4d77bc090d, State: Initialized, Role: FOLLOWER
I20260927 13:52:37.923604 2142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } }
I20260927 13:52:37.930781 2262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } } }
I20260927 13:52:37.931243 2262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:52:37.933313 2142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:52:37.933357 2117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e055b2928fb4437e98cb0c809cd15172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" is_pre_election: true
I20260927 13:52:37.934022 2117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e055b2928fb4437e98cb0c809cd15172 in term 0.
I20260927 13:52:37.935118 2116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e055b2928fb4437e98cb0c809cd15172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c3df47f4ae9402eb546ff4d77bc090d"
I20260927 13:52:37.935484 2116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:37.939059 2116 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e055b2928fb4437e98cb0c809cd15172 in term 1.
I20260927 13:52:37.941157 2117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [term 1 FOLLOWER]: Refusing update from remote peer e055b2928fb4437e98cb0c809cd15172: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:52:37.942235 2258 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:52:37.949892 2043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [term 1 FOLLOWER]: Refusing update from remote peer e055b2928fb4437e98cb0c809cd15172: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:52:37.950875 2258 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:52:37.983901 2244 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e055b2928fb4437e98cb0c809cd15172. Latest consensus state: current_term: 1 leader_uuid: "e055b2928fb4437e98cb0c809cd15172" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } } }
I20260927 13:52:37.984267 2244 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:52:38.006594 2267 mvcc.cc:205] Tried to move back new op lower bound from 7333958278901194752 to 7333958278741970944. Current Snapshot: MvccSnapshot[applied={T|T < 7333958278901194752}]
I20260927 13:52:38.011080 2262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e055b2928fb4437e98cb0c809cd15172. Latest consensus state: current_term: 1 leader_uuid: "e055b2928fb4437e98cb0c809cd15172" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } } }
I20260927 13:52:38.011454 2262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:52:38.018299 2279 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 13:52:38.020100 2265 mvcc.cc:205] Tried to move back new op lower bound from 7333958278901194752 to 7333958278741970944. Current Snapshot: MvccSnapshot[applied={T|T < 7333958278901194752}]
I20260927 13:52:38.025756 2244 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e055b2928fb4437e98cb0c809cd15172" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } } }
I20260927 13:52:38.026011 2244 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:52:38.029364 2258 sys_catalog.cc:455] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e055b2928fb4437e98cb0c809cd15172" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } } }
I20260927 13:52:38.029691 2258 sys_catalog.cc:458] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:38.029932 2258 sys_catalog.cc:455] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e055b2928fb4437e98cb0c809cd15172" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } } }
I20260927 13:52:38.030190 2258 sys_catalog.cc:458] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:38.031554 2260 catalog_manager.cc:1347] Generated new cluster ID: 8696c1986f14441e85c6d3447cfa25e1
I20260927 13:52:38.031677 2260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:52:38.037263 2262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e055b2928fb4437e98cb0c809cd15172" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c51301454b47f6b188dc465b6e22c5" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39337 } } peers { permanent_uuid: "3c3df47f4ae9402eb546ff4d77bc090d" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 37985 } } peers { permanent_uuid: "e055b2928fb4437e98cb0c809cd15172" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46135 } } }
I20260927 13:52:38.037554 2262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:52:38.067653 2260 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:52:38.069589 2260 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:52:38.102375 2260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172: Generated new TSK 0
I20260927 13:52:38.103977 2260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:52:38.808703 2255 catalog_manager.cc:1259] Loaded cluster ID: 8696c1986f14441e85c6d3447cfa25e1
I20260927 13:52:38.808815 2255 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5: loading cluster ID for follower catalog manager: success
I20260927 13:52:38.811234 2255 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5: acquiring CA information for follower catalog manager: success
I20260927 13:52:38.813102 2255 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f3c51301454b47f6b188dc465b6e22c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:52:39.022691 2279 catalog_manager.cc:1259] Loaded cluster ID: 8696c1986f14441e85c6d3447cfa25e1
I20260927 13:52:39.022820 2279 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d: loading cluster ID for follower catalog manager: success
I20260927 13:52:39.025573 2279 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d: acquiring CA information for follower catalog manager: success
I20260927 13:52:39.027523 2279 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c3df47f4ae9402eb546ff4d77bc090d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:52:39.189651 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 2001
I20260927 13:52:39.191160 2063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:39.300864 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 2001
I20260927 13:52:39.319459 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 2075
W20260927 13:52:39.319325 2166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.133.126:39337 (error 108)
I20260927 13:52:39.320757 2137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:39.321902 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:39.321013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:39337 (local address 127.0.0.1:56150)
0927 13:52:39.321526 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:39.321675 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":272,"threads_started":1}
W20260927 13:52:39.322142 2166 consensus_peers.cc:487] T 00000000000000000000000000000000 P e055b2928fb4437e98cb0c809cd15172 -> Peer f3c51301454b47f6b188dc465b6e22c5 (127.20.133.126:39337): Couldn't send request to peer f3c51301454b47f6b188dc465b6e22c5. Status: Network error: Client connection negotiation failed: client connection to 127.20.133.126:39337: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 13:52:39.436151 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 2075
I20260927 13:52:39.452598 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 2149
I20260927 13:52:39.453670 2211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:39.562214 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 2149
2026-09-27T13:52:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4741 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:52:39 dist-test-slave-q25r krb5kdc[2295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:52:39 dist-test-slave-q25r krb5kdc[2295](info): set up 2 sockets
Sep 27 13:52:39 dist-test-slave-q25r krb5kdc[2295](info): commencing operation
krb5kdc: starting...
W20260927 13:52:40.593375 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:52:40 dist-test-slave-q25r krb5kdc[2295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:52:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:52:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:52:40.868534 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:36193
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:36193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:52:41.263311 2308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:41.263696 2308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:41.263818 2308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:41.272029 2308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:52:41.272195 2308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:41.272249 2308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:41.272369 2308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:52:41.272439 2308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:52:41.283893 2308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:36193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:41.285681 2308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:41.287300 2308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:41.300904 2316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:41.300953 2314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:41.301195 2313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:41.302037 2308 server_base.cc:1034] running on GCE node
I20260927 13:52:41.302968 2308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:41.304842 2308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:41.306088 2308 hybrid_clock.cc:648] HybridClock initialized: now 1790517161306037 us; error 50 us; skew 500 ppm
I20260927 13:52:41.306504 2308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:41 dist-test-slave-q25r krb5kdc[2295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517161, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:41.309859 2308 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:52:41.311636 2308 webserver.cc:460] Webserver started at http://127.20.133.126:34401/ using document root <none> and password file <none>
I20260927 13:52:41.312440 2308 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:41.312561 2308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:41.312846 2308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:52:41.316085 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "9f0e766118c84feba1420af40af4edc2"
format_stamp: "Formatted at 2026-09-27 13:52:41 on dist-test-slave-q25r"
server_key: "2f4fb7f11baf71d1367e4ed3aae1d378"
server_key_iv: "887b3c8c6104100213f876e4e59645d6"
server_key_version: "encryptionkey@0"
I20260927 13:52:41.316905 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "9f0e766118c84feba1420af40af4edc2"
format_stamp: "Formatted at 2026-09-27 13:52:41 on dist-test-slave-q25r"
server_key: "2f4fb7f11baf71d1367e4ed3aae1d378"
server_key_iv: "887b3c8c6104100213f876e4e59645d6"
server_key_version: "encryptionkey@0"
I20260927 13:52:41.323014 2308 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 13:52:41.327606 2323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:41.329845 2308 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:52:41.330098 2308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "9f0e766118c84feba1420af40af4edc2"
format_stamp: "Formatted at 2026-09-27 13:52:41 on dist-test-slave-q25r"
server_key: "2f4fb7f11baf71d1367e4ed3aae1d378"
server_key_iv: "887b3c8c6104100213f876e4e59645d6"
server_key_version: "encryptionkey@0"
I20260927 13:52:41.330314 2308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:41.346684 2308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:41.351590 2308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:41.351879 2308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:41.373690 2308 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:36193
I20260927 13:52:41.373786 2374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:36193 every 8 connection(s)
I20260927 13:52:41.375248 2308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:52:41.377039 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 2308
I20260927 13:52:41.377365 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:52:41.378283 21013 external_mini_cluster.cc:1428] Setting key 05659ddb31855bfb1c5464f980cbf952
I20260927 13:52:41.381207 2375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:52:41 dist-test-slave-q25r krb5kdc[2295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:41.397382 2375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2: Bootstrap starting.
I20260927 13:52:41.401999 2375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:41.403424 2375 log.cc:824] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:41.405681 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:41.380615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.20.133.126:36193)
0927 13:52:41.381487 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:41.381500 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:41.381556 (+ 56us) server_negotiation.cc:407] Connection header received
0927 13:52:41.383371 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:41.383398 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:41.383891 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:41.384303 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:41.386629 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:41.388152 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:41.390279 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:41.391112 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:41.391282 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:41.397100 (+ 5818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:41.397109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:41.397129 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:41.397185 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:41.400709 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:41.401576 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:41.401584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:41.401593 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:41.401795 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:41.402180 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:41.402183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:41.402186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:41.402833 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:41.403140 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:41.403673 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 13:52:41.404277 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":379,"threads_started":1}
I20260927 13:52:41.407387 2375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2: No bootstrap required, opened a new log
I20260927 13:52:41.410651 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:52:41.417186 2375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e766118c84feba1420af40af4edc2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36193 } }
I20260927 13:52:41.417711 2375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:41.417822 2375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0e766118c84feba1420af40af4edc2, State: Initialized, Role: FOLLOWER
I20260927 13:52:41.418543 2375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e766118c84feba1420af40af4edc2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36193 } }
I20260927 13:52:41.418794 2375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:52:41.418941 2375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:52:41.419116 2375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:41.421687 2375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e766118c84feba1420af40af4edc2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36193 } }
I20260927 13:52:41.422325 2375 leader_election.cc:302] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f0e766118c84feba1420af40af4edc2; no voters:
I20260927 13:52:41.423085 2375 leader_election.cc:288] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:52:41.423233 2380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:52:41.423966 2380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9f0e766118c84feba1420af40af4edc2, State: Running, Role: LEADER
I20260927 13:52:41.424680 2380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e766118c84feba1420af40af4edc2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36193 } }
I20260927 13:52:41.426080 2375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:52:41.428287 2382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f0e766118c84feba1420af40af4edc2. Latest consensus state: current_term: 1 leader_uuid: "9f0e766118c84feba1420af40af4edc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e766118c84feba1420af40af4edc2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36193 } } }
I20260927 13:52:41.428892 2382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:41.429219 2381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f0e766118c84feba1420af40af4edc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e766118c84feba1420af40af4edc2" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36193 } } }
I20260927 13:52:41.429644 2381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:41.431988 2388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:52:41.438974 2388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:52:41.449553 2388 catalog_manager.cc:1347] Generated new cluster ID: a39a94f0fccc4462bee49bde9c4d9f0f
I20260927 13:52:41.449720 2388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:52:41.470706 2388 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:52:41.472886 2388 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:52:41.481472 2388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f0e766118c84feba1420af40af4edc2: Generated new TSK 0
I20260927 13:52:41.482736 2388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:52:41.525830 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 2308
I20260927 13:52:41.527558 2370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:41.626147 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 2308
2026-09-27T13:52:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2065 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:52:41 dist-test-slave-q25r krb5kdc[2403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:52:41 dist-test-slave-q25r krb5kdc[2403](info): set up 2 sockets
Sep 27 13:52:41 dist-test-slave-q25r krb5kdc[2403](info): commencing operation
krb5kdc: starting...
W20260927 13:52:42.676874 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:52:42 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:52:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:52:42.948154 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:37329
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:37329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:52:43.301128 2416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:43.301474 2416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:43.301607 2416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:43.309062 2416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:52:43.309206 2416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:43.309294 2416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:43.309371 2416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:52:43.309439 2416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:52:43.319990 2416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:37329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:43.321471 2416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:43.322822 2416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:43.335490 2421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:43.335888 2424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:43.335958 2422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:43.337736 2416 server_base.cc:1034] running on GCE node
I20260927 13:52:43.338827 2416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:43.340833 2416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:43.342094 2416 hybrid_clock.cc:648] HybridClock initialized: now 1790517163342036 us; error 58 us; skew 500 ppm
I20260927 13:52:43.342537 2416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:43 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517163, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:43.346342 2416 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:52:43.348308 2416 webserver.cc:460] Webserver started at http://127.20.133.126:46677/ using document root <none> and password file <none>
I20260927 13:52:43.349226 2416 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:43.349357 2416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:43.349684 2416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:52:43.353880 2416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "0bdddcda75c14811bc5945dea24078bf"
format_stamp: "Formatted at 2026-09-27 13:52:43 on dist-test-slave-q25r"
server_key: "5cb6083ddf77d2160305a37895997cae"
server_key_iv: "a93632c78e597cdfc4c90198b58aece1"
server_key_version: "encryptionkey@0"
I20260927 13:52:43.354914 2416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "0bdddcda75c14811bc5945dea24078bf"
format_stamp: "Formatted at 2026-09-27 13:52:43 on dist-test-slave-q25r"
server_key: "5cb6083ddf77d2160305a37895997cae"
server_key_iv: "a93632c78e597cdfc4c90198b58aece1"
server_key_version: "encryptionkey@0"
I20260927 13:52:43.361886 2416 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260927 13:52:43.366341 2431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:43.368157 2416 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 13:52:43.368403 2416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "0bdddcda75c14811bc5945dea24078bf"
format_stamp: "Formatted at 2026-09-27 13:52:43 on dist-test-slave-q25r"
server_key: "5cb6083ddf77d2160305a37895997cae"
server_key_iv: "a93632c78e597cdfc4c90198b58aece1"
server_key_version: "encryptionkey@0"
I20260927 13:52:43.368649 2416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:43.396708 2416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:43.401921 2416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:43.402215 2416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:43.423727 2416 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:37329
I20260927 13:52:43.423780 2482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:37329 every 8 connection(s)
I20260927 13:52:43.425508 2416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:52:43.426381 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 2416
I20260927 13:52:43.426666 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:52:43.427546 21013 external_mini_cluster.cc:1428] Setting key 769c2217f55df83c292f8952bfb35684
I20260927 13:52:43.430962 2483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:52:43 dist-test-slave-q25r krb5kdc[2403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:43.446085 2483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Bootstrap starting.
I20260927 13:52:43.449851 2483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:43.451052 2483 log.cc:824] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:43.455403 2483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: No bootstrap required, opened a new log
I20260927 13:52:43.456848 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:43.430180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38998 (local address 127.20.133.126:37329)
0927 13:52:43.431165 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:43.431181 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:43.431238 (+ 57us) server_negotiation.cc:407] Connection header received
0927 13:52:43.433331 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:43.433364 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:43.433978 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:43.434469 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:43.436589 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:43.438296 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:43.440497 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:43.441358 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:43.441539 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:43.447068 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:43.447079 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:43.447102 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:43.447156 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:43.450989 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:43.452075 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:43.452085 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:43.452093 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:43.452402 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:43.453034 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:43.453044 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:43.453047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:43.453900 (+ 853us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:43.454270 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:43.454862 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 13:52:43.455394 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":415,"threads_started":1}
I20260927 13:52:43.463884 2483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:43.464291 2483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:43.464432 2483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bdddcda75c14811bc5945dea24078bf, State: Initialized, Role: FOLLOWER
I20260927 13:52:43.465039 2483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:43.465258 2483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:52:43.465363 2483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:52:43.465497 2483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:43.467700 2483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:43.468204 2483 leader_election.cc:302] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bdddcda75c14811bc5945dea24078bf; no voters:
I20260927 13:52:43.468933 2483 leader_election.cc:288] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:52:43.469152 2488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:52:43.470357 2488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 1 LEADER]: Becoming Leader. State: Replica: 0bdddcda75c14811bc5945dea24078bf, State: Running, Role: LEADER
I20260927 13:52:43.471177 2488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:43.471451 2483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:52:43.474889 2490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bdddcda75c14811bc5945dea24078bf. Latest consensus state: current_term: 1 leader_uuid: "0bdddcda75c14811bc5945dea24078bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } } }
I20260927 13:52:43.474970 2489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bdddcda75c14811bc5945dea24078bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } } }
I20260927 13:52:43.475514 2490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:43.475515 2489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:43.479027 2495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:52:43.485857 2495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:52:43.498145 2495 catalog_manager.cc:1347] Generated new cluster ID: f47cb84bf3c54b608cd806329fe7f61c
I20260927 13:52:43.498293 2495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:52:43.512112 2495 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:52:43.513965 2495 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:52:43.527585 2495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Generated new TSK 0
I20260927 13:52:43.528885 2495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:52:43.605611 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37329
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:52:43.978967 2509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:43.979349 2509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:43.979518 2509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:43.987407 2509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:43.987581 2509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:43.987751 2509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:52:43.999699 2509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:44.001257 2509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:44.002684 2509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:44.016571 2514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:44.016601 2515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:44.016601 2517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:44.017426 2509 server_base.cc:1034] running on GCE node
I20260927 13:52:44.018020 2509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:44.019040 2509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:44.020272 2509 hybrid_clock.cc:648] HybridClock initialized: now 1790517164020218 us; error 42 us; skew 500 ppm
I20260927 13:52:44.020649 2509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:44 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517164, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:44.024117 2509 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:52:44.025949 2509 webserver.cc:460] Webserver started at http://127.20.133.65:40981/ using document root <none> and password file <none>
I20260927 13:52:44.026923 2509 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:44.027050 2509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:44.027380 2509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:52:44.030701 2509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "811ee5aef57b441293e37ab5f4847587"
format_stamp: "Formatted at 2026-09-27 13:52:44 on dist-test-slave-q25r"
server_key: "2c4e227fadd269977b43383656d0c3c7"
server_key_iv: "b3233d2ac78aea5b0c888a69fc6dde30"
server_key_version: "encryptionkey@0"
I20260927 13:52:44.031538 2509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "811ee5aef57b441293e37ab5f4847587"
format_stamp: "Formatted at 2026-09-27 13:52:44 on dist-test-slave-q25r"
server_key: "2c4e227fadd269977b43383656d0c3c7"
server_key_iv: "b3233d2ac78aea5b0c888a69fc6dde30"
server_key_version: "encryptionkey@0"
I20260927 13:52:44.037380 2509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 13:52:44.041273 2524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:44.042917 2509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:52:44.043107 2509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "811ee5aef57b441293e37ab5f4847587"
format_stamp: "Formatted at 2026-09-27 13:52:44 on dist-test-slave-q25r"
server_key: "2c4e227fadd269977b43383656d0c3c7"
server_key_iv: "b3233d2ac78aea5b0c888a69fc6dde30"
server_key_version: "encryptionkey@0"
I20260927 13:52:44.043344 2509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:44.070042 2509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:44.074738 2509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:44.075024 2509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:44.076084 2509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:44.077466 2509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:44.077545 2509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:44.077656 2509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:44.077706 2509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:44.107712 2509 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:45131
I20260927 13:52:44.107752 2636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:45131 every 8 connection(s)
I20260927 13:52:44.108983 2509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:52:44.116911 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 2509
I20260927 13:52:44.117129 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:52:44.117913 21013 external_mini_cluster.cc:1428] Setting key 0664085587f843bd5169121c7cfae9ed
Sep 27 13:52:44 dist-test-slave-q25r krb5kdc[2403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517164, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:44.130578 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:44.111585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:57233 (local address 127.20.133.126:37329)
0927 13:52:44.112068 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:44.112074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:44.112904 (+ 830us) server_negotiation.cc:407] Connection header received
0927 13:52:44.114758 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:44.114763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:44.114845 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:44.114982 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:44.116508 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:44.117581 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:44.119539 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:44.120058 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:44.120186 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:44.123716 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:44.123722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:44.123726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:44.123766 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:44.126869 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:44.127751 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:44.127756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:44.127759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:44.128137 (+ 378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:44.128795 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:44.128801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:44.128804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:44.129114 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:44.129234 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:44.130131 (+ 897us) server_negotiation.cc:299] Negotiation successful
0927 13:52:44.130321 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260927 13:52:44.131978 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:44.112006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:57233)
0927 13:52:44.112752 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:44.112790 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:44.114432 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:44.115166 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:44.115175 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:44.115701 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:44.116276 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:44.116294 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:44.117784 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:44.117788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:44.119379 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:44.119399 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:44.120277 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:44.120282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:44.120553 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:44.121196 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:44.121220 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:44.123439 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:44.127073 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:44.127082 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:44.127097 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:44.127609 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:44.128319 (+ 710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:44.128325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:44.128328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:44.128642 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:44.129265 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:44.129274 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:44.129726 (+ 452us) client_negotiation.cc:769] Sending connection context
0927 13:52:44.130042 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 13:52:44.130595 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":300,"threads_started":1}
I20260927 13:52:44.133654 2637 heartbeater.cc:347] Connected to a master server at 127.20.133.126:37329
I20260927 13:52:44.133945 2637 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:44.134683 2637 heartbeater.cc:511] Master 127.20.133.126:37329 requested a full tablet report, sending...
I20260927 13:52:44.136984 2448 ts_manager.cc:194] Registered new tserver with Master: 811ee5aef57b441293e37ab5f4847587 (127.20.133.65:45131)
I20260927 13:52:44.139030 2448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:57233
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:52:44.171592 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37329
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:52:44.516007 2643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:44.516389 2643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:44.516558 2643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:44.524121 2643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:44.524245 2643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:44.524420 2643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:52:44.534804 2643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:44.536149 2643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:44.537670 2643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:44.550513 2648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:44.550601 2651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:44.550570 2649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:44.551441 2643 server_base.cc:1034] running on GCE node
I20260927 13:52:44.552091 2643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:44.553212 2643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:44.554462 2643 hybrid_clock.cc:648] HybridClock initialized: now 1790517164554399 us; error 57 us; skew 500 ppm
I20260927 13:52:44.554840 2643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:44 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517164, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:44.558102 2643 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:52:44.559521 2643 webserver.cc:460] Webserver started at http://127.20.133.66:36867/ using document root <none> and password file <none>
I20260927 13:52:44.560233 2643 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:44.560374 2643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:44.560678 2643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:52:44.563720 2643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "1bd3d760c0ea4719a54d92c3a25100af"
format_stamp: "Formatted at 2026-09-27 13:52:44 on dist-test-slave-q25r"
server_key: "bf3a3b3a340a7d4635210f2f7193275e"
server_key_iv: "268a18db9c47cf4488217dc8a3e3a5a4"
server_key_version: "encryptionkey@0"
I20260927 13:52:44.564559 2643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "1bd3d760c0ea4719a54d92c3a25100af"
format_stamp: "Formatted at 2026-09-27 13:52:44 on dist-test-slave-q25r"
server_key: "bf3a3b3a340a7d4635210f2f7193275e"
server_key_iv: "268a18db9c47cf4488217dc8a3e3a5a4"
server_key_version: "encryptionkey@0"
I20260927 13:52:44.569875 2643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:52:44.573844 2658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:44.575368 2643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:52:44.575556 2643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "1bd3d760c0ea4719a54d92c3a25100af"
format_stamp: "Formatted at 2026-09-27 13:52:44 on dist-test-slave-q25r"
server_key: "bf3a3b3a340a7d4635210f2f7193275e"
server_key_iv: "268a18db9c47cf4488217dc8a3e3a5a4"
server_key_version: "encryptionkey@0"
I20260927 13:52:44.575770 2643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:44.590682 2643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:44.595048 2643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:44.595331 2643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:44.596357 2643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:44.597810 2643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:44.597896 2643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:44.598021 2643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:44.598078 2643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:44.629630 2643 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:44463
I20260927 13:52:44.629678 2770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:44463 every 8 connection(s)
I20260927 13:52:44.630970 2643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:52:44.634963 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 2643
I20260927 13:52:44.635166 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:52:44.635592 21013 external_mini_cluster.cc:1428] Setting key 951011101e20576c1f0b25055bb90d74
Sep 27 13:52:44 dist-test-slave-q25r krb5kdc[2403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517164, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
I20260927 13:52:44.658700 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:44.633653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:55409 (local address 127.20.133.126:37329)
0927 13:52:44.634091 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:44.634096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:44.634851 (+ 755us) server_negotiation.cc:407] Connection header received
0927 13:52:44.636938 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:44.636943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:44.637009 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:44.637113 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:44.638777 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:44.639498 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:44.641124 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:44.641549 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:44.641660 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:44.650517 (+ 8857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:44.650525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:44.650530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:44.650579 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:44.654012 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:44.655107 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:44.655114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:44.655118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:44.655285 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:44.655891 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:44.655896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:44.655900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:44.656293 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:44.656479 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:44.657946 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0927 13:52:44.658196 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":228,"threads_started":1}
I20260927 13:52:44.659894 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:44.633989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:55409)
0927 13:52:44.634688 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:44.634728 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:44.636654 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:44.637396 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:44.637408 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:44.638026 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:44.638600 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:44.638619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:44.639664 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:44.639668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:44.640973 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:44.640987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:44.641778 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:44.641785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:44.642037 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:44.646937 (+ 4900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:44.646972 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:44.650150 (+ 3178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:44.654205 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:44.654216 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:44.654239 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:44.654934 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:44.655422 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:44.655444 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:44.655447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:44.655758 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:44.656567 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:44.656578 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:44.657085 (+ 507us) client_negotiation.cc:769] Sending connection context
0927 13:52:44.657795 (+ 710us) client_negotiation.cc:240] Negotiation successful
0927 13:52:44.658410 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":259,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
I20260927 13:52:44.664857 2771 heartbeater.cc:347] Connected to a master server at 127.20.133.126:37329
I20260927 13:52:44.665205 2771 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:44.666038 2771 heartbeater.cc:511] Master 127.20.133.126:37329 requested a full tablet report, sending...
I20260927 13:52:44.667713 2448 ts_manager.cc:194] Registered new tserver with Master: 1bd3d760c0ea4719a54d92c3a25100af (127.20.133.66:44463)
I20260927 13:52:44.668784 2448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:55409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:52:44.693411 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37329
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:52:45.025274 2777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:45.025588 2777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:45.025701 2777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:45.033215 2777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:45.033357 2777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:45.033507 2777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:52:45.043643 2777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:45.045063 2777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:45.046375 2777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:45.061065 2783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:45.061152 2782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:45.061076 2785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:45.062919 2777 server_base.cc:1034] running on GCE node
I20260927 13:52:45.063640 2777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:45.064774 2777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:45.066000 2777 hybrid_clock.cc:648] HybridClock initialized: now 1790517165065913 us; error 81 us; skew 500 ppm
I20260927 13:52:45.066339 2777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:45 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517165, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:45.069537 2777 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:52:45.071620 2777 webserver.cc:460] Webserver started at http://127.20.133.67:39309/ using document root <none> and password file <none>
I20260927 13:52:45.072404 2777 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:45.072546 2777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:45.072878 2777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:52:45.076027 2777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "294c2c702597464594e70c7ef56bd642"
format_stamp: "Formatted at 2026-09-27 13:52:45 on dist-test-slave-q25r"
server_key: "297d5b312720e92acac848ae1c71e532"
server_key_iv: "bb389c1f635d7c9115a2adda8cb25172"
server_key_version: "encryptionkey@0"
I20260927 13:52:45.077025 2777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "294c2c702597464594e70c7ef56bd642"
format_stamp: "Formatted at 2026-09-27 13:52:45 on dist-test-slave-q25r"
server_key: "297d5b312720e92acac848ae1c71e532"
server_key_iv: "bb389c1f635d7c9115a2adda8cb25172"
server_key_version: "encryptionkey@0"
I20260927 13:52:45.083098 2777 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 13:52:45.087358 2792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:45.089177 2777 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 13:52:45.089398 2777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "294c2c702597464594e70c7ef56bd642"
format_stamp: "Formatted at 2026-09-27 13:52:45 on dist-test-slave-q25r"
server_key: "297d5b312720e92acac848ae1c71e532"
server_key_iv: "bb389c1f635d7c9115a2adda8cb25172"
server_key_version: "encryptionkey@0"
I20260927 13:52:45.089630 2777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:45.120543 2777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:45.125214 2777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:45.125492 2777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:45.126564 2777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:45.128412 2777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:45.128552 2777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 13:52:45.128698 2777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:45.128757 2777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:45.142665 2637 heartbeater.cc:503] Master 127.20.133.126:37329 was elected leader, sending a full tablet report...
I20260927 13:52:45.165400 2777 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:43781
I20260927 13:52:45.165480 2904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:43781 every 8 connection(s)
I20260927 13:52:45.166781 2777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:52:45.168021 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 2777
I20260927 13:52:45.168259 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:52:45.168907 21013 external_mini_cluster.cc:1428] Setting key 0357711b0d0ac300e0e26284365bcf18
Sep 27 13:52:45 dist-test-slave-q25r krb5kdc[2403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517165, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:45.190577 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.170309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:36595 (local address 127.20.133.126:37329)
0927 13:52:45.170820 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:45.170827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:45.171897 (+ 1070us) server_negotiation.cc:407] Connection header received
0927 13:52:45.174016 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:45.174020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:45.174103 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:45.174314 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:45.175900 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.176842 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.178650 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.179100 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.179328 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.183469 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:45.183478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:45.183483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:45.183534 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:45.187115 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:45.188010 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:45.188015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:45.188018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:45.188166 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:45.188743 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:45.188747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:45.188749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:45.189075 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:45.189222 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:45.190033 (+ 811us) server_negotiation.cc:299] Negotiation successful
0927 13:52:45.190247 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":223,"threads_started":1}
I20260927 13:52:45.191565 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.170861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:36595)
0927 13:52:45.171719 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:45.171775 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:45.173704 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:45.174548 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:45.174557 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:45.175090 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:45.175706 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.175726 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.177030 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.177034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.178466 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.178484 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.179477 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.179483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.179766 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.180655 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:45.180695 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:45.183154 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:45.187291 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:45.187300 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:45.187317 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:45.187859 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:45.188310 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:45.188314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:45.188317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:45.188613 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:45.189222 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:45.189228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:45.189629 (+ 401us) client_negotiation.cc:769] Sending connection context
0927 13:52:45.189881 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 13:52:45.190327 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":320,"threads_started":1}
I20260927 13:52:45.193547 2905 heartbeater.cc:347] Connected to a master server at 127.20.133.126:37329
I20260927 13:52:45.193984 2905 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:45.194984 2905 heartbeater.cc:511] Master 127.20.133.126:37329 requested a full tablet report, sending...
I20260927 13:52:45.196933 2448 ts_manager.cc:194] Registered new tserver with Master: 294c2c702597464594e70c7ef56bd642 (127.20.133.67:43781)
I20260927 13:52:45.198028 2448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:36595
I20260927 13:52:45.199187 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:52:45.218219 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.204115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.20.133.126:37329)
0927 13:52:45.204466 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:45.204473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:45.204816 (+ 343us) server_negotiation.cc:407] Connection header received
0927 13:52:45.204994 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:45.204999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:45.205086 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:45.205178 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:45.206575 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.207410 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.208948 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.209402 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.209509 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.212211 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:45.212216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:45.212220 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:45.212258 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:45.215143 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:45.216118 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:45.216122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:45.216125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:45.216271 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:45.216706 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:45.216709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:45.216711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:45.217096 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:45.217311 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:45.217705 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 13:52:45.217923 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 13:52:45.223631 2448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:52:45.231463 2448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:52:45.272306 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.247907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.20.133.66:44463)
0927 13:52:45.261660 (+ 13753us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:45.261671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:45.261719 (+ 48us) server_negotiation.cc:407] Connection header received
0927 13:52:45.261861 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:45.261870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:45.262179 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:45.262472 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:45.264757 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.266328 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.269110 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.270309 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.270460 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.270727 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:45.271592 (+ 865us) server_negotiation.cc:299] Negotiation successful
0927 13:52:45.271854 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13568,"thread_start_us":13401,"threads_started":1}
I20260927 13:52:45.273205 2919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.247931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:45131 (local address 127.0.0.1:37152)
0927 13:52:45.265063 (+ 17132us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:45.265092 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:45.265249 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:45.265965 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:45.265970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:45.265997 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:45.266377 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.266389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.268172 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.268178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.270298 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.270308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.272121 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.272127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.272320 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.272387 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 13:52:45.272576 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 13:52:45.272728 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17011,"mutex_wait_us":25,"thread_start_us":14919,"threads_started":1}
I20260927 13:52:45.273837 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.248276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.20.133.65:45131)
0927 13:52:45.262912 (+ 14636us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:45.262920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:45.265225 (+ 2305us) server_negotiation.cc:407] Connection header received
0927 13:52:45.265422 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:45.265431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:45.265727 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:45.265978 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:45.266562 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.267986 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.270446 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.271573 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.271709 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.271956 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:45.273434 (+ 1478us) server_negotiation.cc:299] Negotiation successful
0927 13:52:45.273576 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14371,"thread_start_us":14594,"threads_started":1}
I20260927 13:52:45.274065 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.247352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.20.133.67:43781)
0927 13:52:45.252660 (+ 5308us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:45.252674 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:45.263520 (+ 10846us) server_negotiation.cc:407] Connection header received
0927 13:52:45.263770 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:45.263779 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:45.264092 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:45.264362 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:45.265553 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.267195 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.270127 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.271722 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.271903 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.272151 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:45.273519 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0927 13:52:45.273747 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5087,"thread_start_us":4899,"threads_started":1}
I20260927 13:52:45.273588 2918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.247254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:44463 (local address 127.0.0.1:34222)
0927 13:52:45.260891 (+ 13637us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:45.260966 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:45.261336 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:45.263812 (+ 2476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:45.263822 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:45.263856 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:45.264471 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.264490 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.266538 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.266543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.268934 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.268948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.270864 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.270871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.271140 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.271211 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 13:52:45.271412 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 13:52:45.273134 (+ 1722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13159,"mutex_wait_us":29,"thread_start_us":12910,"threads_started":1}
I20260927 13:52:45.274478 2921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.247277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:43781 (local address 127.0.0.1:58644)
0927 13:52:45.263331 (+ 16054us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:45.263359 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:45.263569 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:45.264691 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:45.264697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:45.264726 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:45.265303 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.265318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.267413 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.267420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.269909 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.269927 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.272387 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.272394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.272574 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.272609 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:52:45.272725 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 13:52:45.272892 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15833,"mutex_wait_us":3,"thread_start_us":17474,"threads_started":1}
I20260927 13:52:45.279134 2839 tablet_service.cc:1467] Processing CreateTablet for tablet c807f299f39d423e88fbcd2dfcc1a3f3 (DEFAULT_TABLE table=test-table [id=9f0e25f588cd40989a17d1f1443b254e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:52:45.279140 2706 tablet_service.cc:1467] Processing CreateTablet for tablet c807f299f39d423e88fbcd2dfcc1a3f3 (DEFAULT_TABLE table=test-table [id=9f0e25f588cd40989a17d1f1443b254e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:52:45.279450 2572 tablet_service.cc:1467] Processing CreateTablet for tablet c807f299f39d423e88fbcd2dfcc1a3f3 (DEFAULT_TABLE table=test-table [id=9f0e25f588cd40989a17d1f1443b254e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:52:45.280850 2706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c807f299f39d423e88fbcd2dfcc1a3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:45.280860 2839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c807f299f39d423e88fbcd2dfcc1a3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:45.281275 2572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c807f299f39d423e88fbcd2dfcc1a3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:45.295200 2924 tablet_bootstrap.cc:492] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: Bootstrap starting.
I20260927 13:52:45.297735 2925 tablet_bootstrap.cc:492] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: Bootstrap starting.
I20260927 13:52:45.299820 2924 tablet_bootstrap.cc:654] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:45.301224 2926 tablet_bootstrap.cc:492] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: Bootstrap starting.
I20260927 13:52:45.302482 2925 tablet_bootstrap.cc:654] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:45.302677 2924 log.cc:824] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:45.304033 2925 log.cc:824] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:45.305305 2926 tablet_bootstrap.cc:654] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:45.306427 2924 tablet_bootstrap.cc:492] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: No bootstrap required, opened a new log
I20260927 13:52:45.306505 2926 log.cc:824] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:45.306763 2924 ts_tablet_manager.cc:1397] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260927 13:52:45.307787 2925 tablet_bootstrap.cc:492] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: No bootstrap required, opened a new log
I20260927 13:52:45.308117 2925 ts_tablet_manager.cc:1397] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 13:52:45.310046 2926 tablet_bootstrap.cc:492] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: No bootstrap required, opened a new log
I20260927 13:52:45.310330 2926 ts_tablet_manager.cc:1397] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 13:52:45.316298 2924 raft_consensus.cc:348] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } } peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } }
I20260927 13:52:45.316857 2924 raft_consensus.cc:374] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:45.316969 2924 raft_consensus.cc:729] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811ee5aef57b441293e37ab5f4847587, State: Initialized, Role: FOLLOWER
I20260927 13:52:45.317668 2924 consensus_queue.cc:260] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } } peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } }
I20260927 13:52:45.318897 2925 raft_consensus.cc:348] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } } peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } }
I20260927 13:52:45.319434 2925 raft_consensus.cc:374] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:45.319542 2925 raft_consensus.cc:729] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294c2c702597464594e70c7ef56bd642, State: Initialized, Role: FOLLOWER
I20260927 13:52:45.320286 2925 consensus_queue.cc:260] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } } peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } }
I20260927 13:52:45.320667 2926 raft_consensus.cc:348] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } } peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } }
I20260927 13:52:45.321099 2926 raft_consensus.cc:374] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:45.321194 2926 raft_consensus.cc:729] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd3d760c0ea4719a54d92c3a25100af, State: Initialized, Role: FOLLOWER
I20260927 13:52:45.321408 2924 ts_tablet_manager.cc:1428] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 13:52:45.321843 2926 consensus_queue.cc:260] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } } peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } }
I20260927 13:52:45.323515 2925 ts_tablet_manager.cc:1428] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260927 13:52:45.323715 2905 heartbeater.cc:503] Master 127.20.133.126:37329 was elected leader, sending a full tablet report...
I20260927 13:52:45.324729 2771 heartbeater.cc:503] Master 127.20.133.126:37329 was elected leader, sending a full tablet report...
I20260927 13:52:45.325300 2926 ts_tablet_manager.cc:1428] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
W20260927 13:52:45.361670 2638 tablet.cc:2369] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:52:45.383133 2772 tablet.cc:2369] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:52:45.418771 2906 tablet.cc:2369] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:52:45.536412 2931 raft_consensus.cc:482] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:52:45.536840 2931 raft_consensus.cc:504] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } } peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } }
I20260927 13:52:45.538960 2931 leader_election.cc:288] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bd3d760c0ea4719a54d92c3a25100af (127.20.133.66:44463), 811ee5aef57b441293e37ab5f4847587 (127.20.133.65:45131)
I20260927 13:52:45.546432 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.539676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:51731 (local address 127.20.133.65:45131)
0927 13:52:45.539906 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:45.539912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:45.539936 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:52:45.540280 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:45.540287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:45.540423 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:45.540582 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:45.541119 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.542112 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.544200 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.545430 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.545577 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.545645 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:45.546042 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 13:52:45.546163 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 13:52:45.546626 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.539404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:45131 (local address 127.20.133.67:51731)
0927 13:52:45.539815 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:45.539839 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:45.540023 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:45.540624 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:45.540628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:45.540700 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:45.540934 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.540948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.542274 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.542281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.544013 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.544030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.545634 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.545641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.545847 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.545890 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 13:52:45.546013 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 13:52:45.546169 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":1,"spinlock_wait_cycles":9600}
I20260927 13:52:45.547402 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.539498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:44463 (local address 127.20.133.67:44585)
0927 13:52:45.540559 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:45.540581 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:45.540765 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:45.541151 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:45.541156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:45.541184 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:45.541616 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.541630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.542997 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.543001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.544941 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:45.544958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.546492 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.546499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:45.546679 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.546712 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:52:45.546819 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 13:52:45.546991 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":47,"spinlock_wait_cycles":41728,"thread_start_us":659,"threads_started":1}
I20260927 13:52:45.547542 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.539615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:44585 (local address 127.20.133.66:44463)
0927 13:52:45.539881 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:45.539888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:45.540725 (+ 837us) server_negotiation.cc:407] Connection header received
0927 13:52:45.540925 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:45.540930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:45.541021 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:45.541119 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:45.541814 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.542792 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.545161 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.546217 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.546390 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.546465 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:45.547144 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 13:52:45.547289 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 13:52:45.548166 2592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c807f299f39d423e88fbcd2dfcc1a3f3" candidate_uuid: "294c2c702597464594e70c7ef56bd642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811ee5aef57b441293e37ab5f4847587" is_pre_election: true
I20260927 13:52:45.548753 2726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c807f299f39d423e88fbcd2dfcc1a3f3" candidate_uuid: "294c2c702597464594e70c7ef56bd642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd3d760c0ea4719a54d92c3a25100af" is_pre_election: true
I20260927 13:52:45.548835 2592 raft_consensus.cc:2454] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294c2c702597464594e70c7ef56bd642 in term 0.
I20260927 13:52:45.549270 2726 raft_consensus.cc:2454] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294c2c702597464594e70c7ef56bd642 in term 0.
I20260927 13:52:45.549646 2793 leader_election.cc:302] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294c2c702597464594e70c7ef56bd642, 811ee5aef57b441293e37ab5f4847587; no voters:
I20260927 13:52:45.550119 2931 raft_consensus.cc:2781] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:52:45.550268 2931 raft_consensus.cc:482] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:52:45.550375 2931 raft_consensus.cc:3037] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:45.553018 2931 raft_consensus.cc:504] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } } peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } }
I20260927 13:52:45.553815 2931 leader_election.cc:288] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [CANDIDATE]: Term 1 election: Requested vote from peers 1bd3d760c0ea4719a54d92c3a25100af (127.20.133.66:44463), 811ee5aef57b441293e37ab5f4847587 (127.20.133.65:45131)
I20260927 13:52:45.554523 2726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c807f299f39d423e88fbcd2dfcc1a3f3" candidate_uuid: "294c2c702597464594e70c7ef56bd642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd3d760c0ea4719a54d92c3a25100af"
I20260927 13:52:45.554522 2592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c807f299f39d423e88fbcd2dfcc1a3f3" candidate_uuid: "294c2c702597464594e70c7ef56bd642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811ee5aef57b441293e37ab5f4847587"
I20260927 13:52:45.554759 2726 raft_consensus.cc:3037] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:45.554798 2592 raft_consensus.cc:3037] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:45.557340 2592 raft_consensus.cc:2454] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294c2c702597464594e70c7ef56bd642 in term 1.
I20260927 13:52:45.557422 2726 raft_consensus.cc:2454] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294c2c702597464594e70c7ef56bd642 in term 1.
I20260927 13:52:45.558003 2794 leader_election.cc:302] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd3d760c0ea4719a54d92c3a25100af, 294c2c702597464594e70c7ef56bd642; no voters:
I20260927 13:52:45.558389 2931 raft_consensus.cc:2781] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:52:45.558971 2931 raft_consensus.cc:686] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 1 LEADER]: Becoming Leader. State: Replica: 294c2c702597464594e70c7ef56bd642, State: Running, Role: LEADER
I20260927 13:52:45.559607 2931 consensus_queue.cc:237] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } } peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } }
I20260927 13:52:45.565153 2445 catalog_manager.cc:5495] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294c2c702597464594e70c7ef56bd642 (127.20.133.67). New cstate: current_term: 1 leader_uuid: "294c2c702597464594e70c7ef56bd642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } health_report { overall_health: UNKNOWN } } }
I20260927 13:52:45.648277 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.642249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.20.133.67:43781)
0927 13:52:45.642443 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:45.642448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:45.642470 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:52:45.642621 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:45.642625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:45.642729 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:45.642890 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:52:45.643317 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.644088 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.645688 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.646151 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.646314 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.646951 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:52:45.647635 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:52:45.647792 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:45.647922 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 13:52:45.648002 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:52:45.659045 2592 raft_consensus.cc:1261] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587 [term 1 FOLLOWER]: Refusing update from remote peer 294c2c702597464594e70c7ef56bd642: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:52:45.659106 2726 raft_consensus.cc:1261] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af [term 1 FOLLOWER]: Refusing update from remote peer 294c2c702597464594e70c7ef56bd642: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:52:45.660225 2931 consensus_queue.cc:1035] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [LEADER]: Connected to new peer: Peer: permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:52:45.660708 2934 consensus_queue.cc:1035] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:52:45.676741 2939 mvcc.cc:205] Tried to move back new op lower bound from 7333958310524637184 to 7333958310137647104. Current Snapshot: MvccSnapshot[applied={T|T < 7333958310524637184}]
I20260927 13:52:45.677800 2937 mvcc.cc:205] Tried to move back new op lower bound from 7333958310524637184 to 7333958310137647104. Current Snapshot: MvccSnapshot[applied={T|T < 7333958310524637184}]
I20260927 13:52:45.692045 2445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:52:45.694478 2445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:52:45.699101 2445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Sending DeleteTablet for 3 replicas of tablet c807f299f39d423e88fbcd2dfcc1a3f3
I20260927 13:52:45.700899 2706 tablet_service.cc:1514] Processing DeleteTablet for tablet c807f299f39d423e88fbcd2dfcc1a3f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:52:45 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:34222
I20260927 13:52:45.700893 2839 tablet_service.cc:1514] Processing DeleteTablet for tablet c807f299f39d423e88fbcd2dfcc1a3f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:52:45 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:58644
I20260927 13:52:45.701180 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 2509
I20260927 13:52:45.701700 2572 tablet_service.cc:1514] Processing DeleteTablet for tablet c807f299f39d423e88fbcd2dfcc1a3f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:52:45 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:37152
I20260927 13:52:45.702831 2948 tablet_replica.cc:330] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: stopping tablet replica
I20260927 13:52:45.703500 2948 raft_consensus.cc:2229] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:52:45.704190 2948 raft_consensus.cc:2258] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:52:45.707026 2948 ts_tablet_manager.cc:1905] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:52:45.711676 2946 tablet_replica.cc:330] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: stopping tablet replica
I20260927 13:52:45.712195 2946 raft_consensus.cc:2229] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:52:45.713052 2946 raft_consensus.cc:2258] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:52:45.713910 2947 tablet_replica.cc:330] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: stopping tablet replica
I20260927 13:52:45.714334 2947 raft_consensus.cc:2229] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:52:45.714898 2947 raft_consensus.cc:2258] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:52:45.715592 2948 ts_tablet_manager.cc:1918] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:52:45.715803 2948 log.cc:1196] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/c807f299f39d423e88fbcd2dfcc1a3f3
I20260927 13:52:45.716421 2948 ts_tablet_manager.cc:1939] T c807f299f39d423e88fbcd2dfcc1a3f3 P 811ee5aef57b441293e37ab5f4847587: Deleting consensus metadata
I20260927 13:52:45.717499 2947 ts_tablet_manager.cc:1905] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:52:45.718837 2432 catalog_manager.cc:4841] TS 811ee5aef57b441293e37ab5f4847587 (127.20.133.65:45131): tablet c807f299f39d423e88fbcd2dfcc1a3f3 (table test-table [id=9f0e25f588cd40989a17d1f1443b254e]) successfully deleted
I20260927 13:52:45.722424 2946 ts_tablet_manager.cc:1905] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:52:45.725046 2947 ts_tablet_manager.cc:1918] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:52:45.725396 2947 log.cc:1196] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/c807f299f39d423e88fbcd2dfcc1a3f3
I20260927 13:52:45.726054 2947 ts_tablet_manager.cc:1939] T c807f299f39d423e88fbcd2dfcc1a3f3 P 1bd3d760c0ea4719a54d92c3a25100af: Deleting consensus metadata
Sep 27 13:52:45 dist-test-slave-q25r krb5kdc[2403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:52:45.728358 2433 catalog_manager.cc:4841] TS 1bd3d760c0ea4719a54d92c3a25100af (127.20.133.66:44463): tablet c807f299f39d423e88fbcd2dfcc1a3f3 (table test-table [id=9f0e25f588cd40989a17d1f1443b254e]) successfully deleted
I20260927 13:52:45.730690 2946 ts_tablet_manager.cc:1918] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:52:45.730908 2946 log.cc:1196] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/c807f299f39d423e88fbcd2dfcc1a3f3
I20260927 13:52:45.731496 2946 ts_tablet_manager.cc:1939] T c807f299f39d423e88fbcd2dfcc1a3f3 P 294c2c702597464594e70c7ef56bd642: Deleting consensus metadata
I20260927 13:52:45.733512 2435 catalog_manager.cc:4841] TS 294c2c702597464594e70c7ef56bd642 (127.20.133.67:43781): tablet c807f299f39d423e88fbcd2dfcc1a3f3 (table test-table [id=9f0e25f588cd40989a17d1f1443b254e]) successfully deleted
I20260927 13:52:45.736950 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.702546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.20.133.65:45131)
0927 13:52:45.702768 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:45.702775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:45.713915 (+ 11140us) server_negotiation.cc:407] Connection header received
0927 13:52:45.714356 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:45.714363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:45.714469 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:45.714542 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:45.718122 (+ 3580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.719248 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.721552 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.722233 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.722339 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.729051 (+ 6712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:45.729076 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:45.729097 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:45.729167 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:45.733398 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:45.734384 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:45.734395 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:45.734399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:45.734614 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:45.735065 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:45.735071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:45.735073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:45.735595 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:45.735967 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:45.736254 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 13:52:45.736548 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:52:45.737684 2632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:45.891292 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 2509
I20260927 13:52:45.906751 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 2643
Sep 27 13:52:45 dist-test-slave-q25r krb5kdc[2403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:52:45.923646 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:45.907596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.20.133.66:44463)
0927 13:52:45.907850 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:45.907856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:45.907879 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:52:45.908029 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:45.908035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:45.908128 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:45.908227 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:45.909523 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.910585 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.911950 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:45.912636 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:45.912780 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:45.917150 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:45.917171 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:45.917191 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:45.917249 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:45.920784 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:45.921503 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:45.921512 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:45.921516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:45.921686 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:45.922078 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:45.922082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:45.922084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:45.922470 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:45.922755 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:45.923082 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 13:52:45.923307 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 13:52:45.924314 2766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:46.015214 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 2643
I20260927 13:52:46.036782 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 2777
Sep 27 13:52:46 dist-test-slave-q25r krb5kdc[2403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:52:46.053661 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:46.037786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.20.133.67:43781)
0927 13:52:46.037991 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:46.037997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:46.038020 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:52:46.038087 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:46.038090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:46.038182 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:46.038284 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:46.039537 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:46.040550 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:46.041858 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:46.042330 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:46.042452 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:46.047367 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:46.047381 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:46.047401 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:46.047464 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:46.050663 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:46.051394 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:46.051402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:46.051404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:46.051576 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:46.051955 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:46.051958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:46.051960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:46.052412 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:46.052739 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:46.053032 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 13:52:46.053286 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:52:46.054427 2900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:46.176935 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 2777
I20260927 13:52:46.193050 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 2416
I20260927 13:52:46.194252 2478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:46.310359 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 2416
I20260927 13:52:46.328672 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:37329
--webserver_interface=127.20.133.126
--webserver_port=46677
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:37329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:52:46.676230 2954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:46.676623 2954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:46.676764 2954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:46.684465 2954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:52:46.684607 2954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:46.684686 2954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:46.684764 2954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:52:46.684827 2954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:52:46.695191 2954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:37329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=46677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:46.696668 2954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:46.698038 2954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:46.720674 2961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:46.720642 2960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:46.720714 2963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:46.721400 2954 server_base.cc:1034] running on GCE node
I20260927 13:52:46.722339 2954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:46.724196 2954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:46.725474 2954 hybrid_clock.cc:648] HybridClock initialized: now 1790517166725405 us; error 52 us; skew 500 ppm
I20260927 13:52:46.725962 2954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:46 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517166, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:46.729722 2954 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:52:46.731513 2954 webserver.cc:460] Webserver started at http://127.20.133.126:46677/ using document root <none> and password file <none>
I20260927 13:52:46.732295 2954 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:46.732465 2954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:46.738838 2954 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:52:46.742007 2970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:46.743458 2954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:52:46.743639 2954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "0bdddcda75c14811bc5945dea24078bf"
format_stamp: "Formatted at 2026-09-27 13:52:43 on dist-test-slave-q25r"
server_key: "5cb6083ddf77d2160305a37895997cae"
server_key_iv: "a93632c78e597cdfc4c90198b58aece1"
server_key_version: "encryptionkey@0"
I20260927 13:52:46.744823 2954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:46.777438 2954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:46.781981 2954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:46.782260 2954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:46.803037 2954 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:37329
I20260927 13:52:46.803056 3021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:37329 every 8 connection(s)
I20260927 13:52:46.804729 2954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:52:46.812028 3022 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:52:46.813463 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 2954
I20260927 13:52:46.814267 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:45131
--local_ip_for_outbound_sockets=127.20.133.65
--tserver_master_addrs=127.20.133.126:37329
--webserver_port=40981
--webserver_interface=127.20.133.65
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:52:46.815913 3022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Bootstrap starting.
I20260927 13:52:46.841274 3022 log.cc:824] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:46.859340 3022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:52:46.860052 3022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Bootstrap complete.
I20260927 13:52:46.873766 3022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:46.875068 3022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bdddcda75c14811bc5945dea24078bf, State: Initialized, Role: FOLLOWER
I20260927 13:52:46.875844 3022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:46.876153 3022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:52:46.876287 3022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:52:46.876489 3022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:52:46.882674 3022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:46.883560 3022 leader_election.cc:302] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bdddcda75c14811bc5945dea24078bf; no voters:
I20260927 13:52:46.884296 3022 leader_election.cc:288] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:52:46.884542 3026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:52:46.885706 3026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 2 LEADER]: Becoming Leader. State: Replica: 0bdddcda75c14811bc5945dea24078bf, State: Running, Role: LEADER
I20260927 13:52:46.886315 3026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:46.886724 3022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:52:46.888996 3028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bdddcda75c14811bc5945dea24078bf. Latest consensus state: current_term: 2 leader_uuid: "0bdddcda75c14811bc5945dea24078bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } } }
I20260927 13:52:46.889431 3028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:46.889966 3027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0bdddcda75c14811bc5945dea24078bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } } }
I20260927 13:52:46.890240 3027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:46.892908 3032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:52:46.901283 3032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:52:46.902843 3032 catalog_manager.cc:1259] Loaded cluster ID: f47cb84bf3c54b608cd806329fe7f61c
I20260927 13:52:46.903014 3032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:52:46.906288 3032 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 13:52:46.923882 3032 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 13:52:46.933561 3032 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:52:46.935500 3032 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Loaded TSK: 0
I20260927 13:52:46.937227 3032 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 13:52:47.188419 3024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:47.188715 3024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:47.188833 3024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:47.196390 3024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:47.196512 3024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:47.196656 3024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:52:47.206691 3024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:45131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=40981
--tserver_master_addrs=127.20.133.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:47.207981 3024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:47.209324 3024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:47.222193 3049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:47.222167 3050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:47.222895 3024 server_base.cc:1034] running on GCE node
W20260927 13:52:47.223172 3052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:47.223958 3024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:47.225203 3024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:47.226552 3024 hybrid_clock.cc:648] HybridClock initialized: now 1790517167226491 us; error 45 us; skew 500 ppm
I20260927 13:52:47.226934 3024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:47 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517167, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:47.230979 3024 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:52:47.232918 3024 webserver.cc:460] Webserver started at http://127.20.133.65:40981/ using document root <none> and password file <none>
I20260927 13:52:47.233853 3024 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:47.234022 3024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:47.241303 3024 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 13:52:47.244869 3059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:47.246538 3024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 13:52:47.246728 3024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "811ee5aef57b441293e37ab5f4847587"
format_stamp: "Formatted at 2026-09-27 13:52:44 on dist-test-slave-q25r"
server_key: "2c4e227fadd269977b43383656d0c3c7"
server_key_iv: "b3233d2ac78aea5b0c888a69fc6dde30"
server_key_version: "encryptionkey@0"
I20260927 13:52:47.248258 3024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:47.274976 3024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:47.279070 3024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:47.279302 3024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:47.280213 3024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:47.281728 3024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:47.281821 3024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:47.281916 3024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:47.281983 3024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:47.311862 3024 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:45131
I20260927 13:52:47.311918 3171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:45131 every 8 connection(s)
I20260927 13:52:47.313349 3024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:52:47.321909 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 3024
I20260927 13:52:47.322474 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:44463
--local_ip_for_outbound_sockets=127.20.133.66
--tserver_master_addrs=127.20.133.126:37329
--webserver_port=36867
--webserver_interface=127.20.133.66
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:52:47 dist-test-slave-q25r krb5kdc[2403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517167, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:47.335889 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:47.316055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:56343 (local address 127.20.133.126:37329)
0927 13:52:47.316640 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:47.316651 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:47.317138 (+ 487us) server_negotiation.cc:407] Connection header received
0927 13:52:47.319208 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:47.319220 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:47.319522 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:47.319806 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:47.321150 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:47.322452 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:47.324013 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:47.324726 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:47.324897 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:47.328651 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:47.328659 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:47.328675 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:47.328718 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:47.331386 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:47.332112 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:47.332119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:47.332125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:47.332289 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:47.332863 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:47.332867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:47.332870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:47.333344 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:47.333529 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:47.334272 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 13:52:47.334644 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":271,"threads_started":1}
I20260927 13:52:47.335889 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:47.316190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:56343)
0927 13:52:47.316966 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:47.317014 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:47.318708 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:47.319907 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:47.319914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:47.320450 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:47.320947 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:47.320963 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:47.322620 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:47.322625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:47.323859 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:47.323872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:47.324986 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:47.324995 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:47.325267 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:47.325981 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:47.326012 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:47.328364 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:47.331534 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:47.331544 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:47.331559 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:47.331965 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:47.332421 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:47.332426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:47.332430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:47.332716 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:47.333481 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:47.333490 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:47.333805 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 13:52:47.334033 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 13:52:47.334599 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":334,"threads_started":1}
I20260927 13:52:47.339330 3172 heartbeater.cc:347] Connected to a master server at 127.20.133.126:37329
I20260927 13:52:47.339663 3172 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:47.340433 3172 heartbeater.cc:511] Master 127.20.133.126:37329 requested a full tablet report, sending...
I20260927 13:52:47.343050 2987 ts_manager.cc:194] Registered new tserver with Master: 811ee5aef57b441293e37ab5f4847587 (127.20.133.65:45131)
I20260927 13:52:47.345237 2987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:56343
W20260927 13:52:47.642773 3176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:47.643082 3176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:47.643179 3176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:47.650592 3176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:47.650714 3176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:47.650830 3176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:52:47.661031 3176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:44463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=36867
--tserver_master_addrs=127.20.133.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:47.662285 3176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:47.663671 3176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:47.676926 3182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:47.677078 3181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:47.677078 3184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:47.677744 3176 server_base.cc:1034] running on GCE node
I20260927 13:52:47.678382 3176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:47.679476 3176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:47.680766 3176 hybrid_clock.cc:648] HybridClock initialized: now 1790517167680692 us; error 58 us; skew 500 ppm
I20260927 13:52:47.681108 3176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:47 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517167, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:47.684291 3176 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:52:47.685914 3176 webserver.cc:460] Webserver started at http://127.20.133.66:36867/ using document root <none> and password file <none>
I20260927 13:52:47.686611 3176 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:47.686735 3176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:47.692364 3176 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:52:47.695284 3191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:47.696872 3176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:52:47.697053 3176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "1bd3d760c0ea4719a54d92c3a25100af"
format_stamp: "Formatted at 2026-09-27 13:52:44 on dist-test-slave-q25r"
server_key: "bf3a3b3a340a7d4635210f2f7193275e"
server_key_iv: "268a18db9c47cf4488217dc8a3e3a5a4"
server_key_version: "encryptionkey@0"
I20260927 13:52:47.698038 3176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:47.715039 3176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:47.719398 3176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:47.719661 3176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:47.720675 3176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:47.722007 3176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:47.722090 3176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:47.722189 3176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:47.722256 3176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:47.752254 3176 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:44463
I20260927 13:52:47.752303 3303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:44463 every 8 connection(s)
I20260927 13:52:47.753677 3176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:52:47.755069 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 3176
I20260927 13:52:47.755718 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:43781
--local_ip_for_outbound_sockets=127.20.133.67
--tserver_master_addrs=127.20.133.126:37329
--webserver_port=39309
--webserver_interface=127.20.133.67
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:52:47 dist-test-slave-q25r krb5kdc[2403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517167, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:47.782261 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:47.756202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:40547 (local address 127.20.133.126:37329)
0927 13:52:47.756437 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:47.756443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:47.763400 (+ 6957us) server_negotiation.cc:407] Connection header received
0927 13:52:47.765488 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:47.765494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:47.765581 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:47.765699 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:47.767159 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:47.768154 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:47.770020 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:47.770606 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:47.770734 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:47.774713 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:47.774722 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:47.774727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:47.774777 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:47.778613 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:47.779597 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:47.779604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:47.779607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:47.779778 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:47.780301 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:47.780306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:47.780309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:47.780711 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:47.780860 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:47.781660 (+ 800us) server_negotiation.cc:299] Negotiation successful
0927 13:52:47.781909 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:52:47.783067 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:47.756706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:40547)
0927 13:52:47.763143 (+ 6437us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:47.763224 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:47.765194 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:47.765867 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:47.765875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:47.766389 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:47.766985 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:47.767003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:47.768404 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:47.768410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:47.769831 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:47.769845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:47.770852 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:47.770858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:47.771081 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:47.771777 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:47.771802 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:47.774427 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:47.778857 (+ 4430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:47.778866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:47.778882 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:47.779423 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:47.779936 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:47.779940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:47.779942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:47.780171 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:47.780861 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:47.780868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:47.781267 (+ 399us) client_negotiation.cc:769] Sending connection context
0927 13:52:47.781524 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 13:52:47.781946 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6058,"thread_start_us":5844,"threads_started":1}
I20260927 13:52:47.784510 3304 heartbeater.cc:347] Connected to a master server at 127.20.133.126:37329
I20260927 13:52:47.784853 3304 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:47.785749 3304 heartbeater.cc:511] Master 127.20.133.126:37329 requested a full tablet report, sending...
I20260927 13:52:47.787392 2987 ts_manager.cc:194] Registered new tserver with Master: 1bd3d760c0ea4719a54d92c3a25100af (127.20.133.66:44463)
I20260927 13:52:47.788262 2987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:40547
W20260927 13:52:48.095954 3307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:48.096280 3307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:48.096439 3307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:48.103930 3307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:48.104081 3307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:48.104234 3307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:52:48.114198 3307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:43781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=39309
--tserver_master_addrs=127.20.133.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:48.115536 3307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:48.116878 3307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:48.129580 3313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:48.129676 3315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:48.129601 3312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:48.130661 3307 server_base.cc:1034] running on GCE node
I20260927 13:52:48.131245 3307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:48.132153 3307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:48.133368 3307 hybrid_clock.cc:648] HybridClock initialized: now 1790517168133316 us; error 42 us; skew 500 ppm
I20260927 13:52:48.133682 3307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:48 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:48.136694 3307 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:52:48.138137 3307 webserver.cc:460] Webserver started at http://127.20.133.67:39309/ using document root <none> and password file <none>
I20260927 13:52:48.138801 3307 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:48.138931 3307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:48.144275 3307 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:52:48.146973 3322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:48.148278 3307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:52:48.148473 3307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "294c2c702597464594e70c7ef56bd642"
format_stamp: "Formatted at 2026-09-27 13:52:45 on dist-test-slave-q25r"
server_key: "297d5b312720e92acac848ae1c71e532"
server_key_iv: "bb389c1f635d7c9115a2adda8cb25172"
server_key_version: "encryptionkey@0"
I20260927 13:52:48.149447 3307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:48.165125 3307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:48.169308 3307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:48.169582 3307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:48.170583 3307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:48.171917 3307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:48.172024 3307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:48.172132 3307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:48.172197 3307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:48.203526 3307 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:43781
I20260927 13:52:48.203605 3434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:43781 every 8 connection(s)
I20260927 13:52:48.205045 3307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:52:48.211409 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 3307
Sep 27 13:52:48 dist-test-slave-q25r krb5kdc[2403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:52:48.227305 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.207712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:50563 (local address 127.20.133.126:37329)
0927 13:52:48.207930 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.207936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.209133 (+ 1197us) server_negotiation.cc:407] Connection header received
0927 13:52:48.210980 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.210985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.211064 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.211170 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:48.213335 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.214278 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.216313 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.216782 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.216867 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.220651 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:48.220659 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:48.220664 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:48.220707 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:48.223591 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:48.224572 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:48.224577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:48.224580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:48.224738 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:48.225239 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:48.225244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:48.225247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:48.225620 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:48.225767 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.226688 (+ 921us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.226936 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:52:48.227566 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.212502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.20.133.126:37329)
0927 13:52:48.213510 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.213515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.213531 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:52:48.213575 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.213579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.213658 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.213764 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:48.215583 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.216304 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.218028 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.218446 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.218558 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.221469 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:48.221476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:48.221480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:48.221533 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:48.224477 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:48.225248 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:48.225252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:48.225254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:48.225418 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:48.225839 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:48.225843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:48.225846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:48.226193 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:48.226457 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.226715 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.227109 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":889,"spinlock_wait_cycles":896,"thread_start_us":765,"threads_started":1}
I20260927 13:52:48.228566 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.208161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:50563)
0927 13:52:48.208948 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:48.208988 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:48.210658 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:48.211482 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:48.211496 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:48.212207 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:52:48.213106 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.213135 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.214457 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.214463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.216167 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.216180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.217004 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.217012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.217320 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.218044 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:52:48.218077 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:52:48.220396 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:52:48.223746 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:48.223756 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:48.223775 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:48.224412 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:48.224882 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:52:48.224885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:52:48.224888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:52:48.225104 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:52:48.225752 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:52:48.225759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:52:48.226229 (+ 470us) client_negotiation.cc:769] Sending connection context
0927 13:52:48.226552 (+ 323us) client_negotiation.cc:240] Negotiation successful
0927 13:52:48.227111 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":328,"threads_started":1}
I20260927 13:52:48.229990 3435 heartbeater.cc:347] Connected to a master server at 127.20.133.126:37329
I20260927 13:52:48.230301 3435 heartbeater.cc:464] Registering TS with master...
I20260927 13:52:48.231010 3435 heartbeater.cc:511] Master 127.20.133.126:37329 requested a full tablet report, sending...
I20260927 13:52:48.232473 2987 ts_manager.cc:194] Registered new tserver with Master: 294c2c702597464594e70c7ef56bd642 (127.20.133.67:43781)
I20260927 13:52:48.233374 2987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:50563
I20260927 13:52:48.243023 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:52:48.252803 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.244356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.20.133.126:37329)
0927 13:52:48.244563 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.244569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.245420 (+ 851us) server_negotiation.cc:407] Connection header received
0927 13:52:48.245578 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.245583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.245799 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.245968 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:52:48.246394 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.247445 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.249770 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.250463 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.250611 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.251280 (+ 669us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:52:48.252078 (+ 798us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:52:48.252328 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.252406 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.252502 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 13:52:48.256685 2987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:52:48.264006 2987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:52:48.302762 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.276594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.20.133.65:45131)
0927 13:52:48.293648 (+ 17054us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.293654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.293692 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:52:48.293788 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.293795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.294069 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.294325 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:48.295424 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.297270 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.300273 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.301649 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.301780 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.301991 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.302280 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.302451 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16903,"thread_start_us":17480,"threads_started":1}
I20260927 13:52:48.303993 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.276308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:45131 (local address 127.0.0.1:37172)
0927 13:52:48.287301 (+ 10993us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:48.287369 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:48.288328 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:48.294626 (+ 6298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:48.294634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:48.294690 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:48.295185 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.295203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.297399 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.297405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.300093 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.300114 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.301861 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.301867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.302054 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.302092 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 13:52:48.302212 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 13:52:48.303545 (+ 1333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10608,"mutex_wait_us":3,"thread_start_us":19750,"threads_started":1}
I20260927 13:52:48.304147 3445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.276981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:44463 (local address 127.0.0.1:34252)
0927 13:52:48.295915 (+ 18934us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:48.295938 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:48.296072 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:48.296707 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:48.296712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:48.296742 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:48.297128 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.297141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.298691 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.298696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.300948 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.300963 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.303256 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.303262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.303449 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.303482 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:52:48.303583 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 13:52:48.303718 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18823,"mutex_wait_us":87,"thread_start_us":9417,"threads_started":1}
I20260927 13:52:48.304422 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.277271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.20.133.67:43781)
0927 13:52:48.294792 (+ 17521us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.294800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.295555 (+ 755us) server_negotiation.cc:407] Connection header received
0927 13:52:48.295662 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.295669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.295903 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.296097 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:48.297047 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.298629 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.301151 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.302512 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.302637 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.302839 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.303916 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.304089 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17360,"thread_start_us":17191,"threads_started":1}
I20260927 13:52:48.304662 3447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.276580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:43781 (local address 127.0.0.1:58674)
0927 13:52:48.295296 (+ 18716us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:48.295317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:48.295437 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:48.296417 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:48.296422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:48.296448 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:48.296809 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.296823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.298806 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.298811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.300958 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.300973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.303233 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.303240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.303562 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.303593 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 13:52:48.303651 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 13:52:48.303759 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18560,"mutex_wait_us":165,"thread_start_us":18419,"threads_started":1}
I20260927 13:52:48.304694 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.277252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.20.133.66:44463)
0927 13:52:48.291598 (+ 14346us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.291609 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.296205 (+ 4596us) server_negotiation.cc:407] Connection header received
0927 13:52:48.296309 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.296317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.296522 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.296676 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:48.297303 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.298277 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.301109 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.302743 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.302867 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.303028 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.303648 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.303817 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14124,"thread_start_us":14385,"threads_started":1}
I20260927 13:52:48.307723 3107 tablet_service.cc:1467] Processing CreateTablet for tablet b21544a9f7e74a9d98b455a0054c11a0 (DEFAULT_TABLE table=test-table [id=ba8b6667291d4387b031bb051a006d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:52:48.307865 3239 tablet_service.cc:1467] Processing CreateTablet for tablet b21544a9f7e74a9d98b455a0054c11a0 (DEFAULT_TABLE table=test-table [id=ba8b6667291d4387b031bb051a006d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:52:48.307865 3370 tablet_service.cc:1467] Processing CreateTablet for tablet b21544a9f7e74a9d98b455a0054c11a0 (DEFAULT_TABLE table=test-table [id=ba8b6667291d4387b031bb051a006d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:52:48.309147 3107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21544a9f7e74a9d98b455a0054c11a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:48.309155 3239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21544a9f7e74a9d98b455a0054c11a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:48.309158 3370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21544a9f7e74a9d98b455a0054c11a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:52:48.325685 3452 tablet_bootstrap.cc:492] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: Bootstrap starting.
I20260927 13:52:48.329419 3452 tablet_bootstrap.cc:654] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:48.330878 3452 log.cc:824] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:48.332226 3451 tablet_bootstrap.cc:492] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af: Bootstrap starting.
I20260927 13:52:48.335038 3452 tablet_bootstrap.cc:492] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: No bootstrap required, opened a new log
I20260927 13:52:48.335291 3452 ts_tablet_manager.cc:1397] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 13:52:48.335548 3453 tablet_bootstrap.cc:492] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587: Bootstrap starting.
I20260927 13:52:48.335839 3451 tablet_bootstrap.cc:654] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:48.337194 3451 log.cc:824] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:48.339321 3453 tablet_bootstrap.cc:654] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587: Neither blocks nor log segments found. Creating new log.
I20260927 13:52:48.340714 3453 log.cc:824] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:48.343772 3453 tablet_bootstrap.cc:492] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587: No bootstrap required, opened a new log
I20260927 13:52:48.344031 3451 tablet_bootstrap.cc:492] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af: No bootstrap required, opened a new log
I20260927 13:52:48.344038 3453 ts_tablet_manager.cc:1397] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260927 13:52:48.343772 3452 raft_consensus.cc:348] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } } peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } }
I20260927 13:52:48.344271 3452 raft_consensus.cc:374] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:48.344305 3451 ts_tablet_manager.cc:1397] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 13:52:48.344435 3452 raft_consensus.cc:729] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294c2c702597464594e70c7ef56bd642, State: Initialized, Role: FOLLOWER
I20260927 13:52:48.345120 3452 consensus_queue.cc:260] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } } peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } }
I20260927 13:52:48.346316 3435 heartbeater.cc:503] Master 127.20.133.126:37329 was elected leader, sending a full tablet report...
I20260927 13:52:48.346968 3452 ts_tablet_manager.cc:1428] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260927 13:52:48.348567 3172 heartbeater.cc:503] Master 127.20.133.126:37329 was elected leader, sending a full tablet report...
I20260927 13:52:48.353801 3451 raft_consensus.cc:348] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } } peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } }
I20260927 13:52:48.354249 3451 raft_consensus.cc:374] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:48.354350 3451 raft_consensus.cc:729] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd3d760c0ea4719a54d92c3a25100af, State: Initialized, Role: FOLLOWER
I20260927 13:52:48.354919 3451 consensus_queue.cc:260] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } } peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } }
I20260927 13:52:48.355371 3453 raft_consensus.cc:348] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } } peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } }
I20260927 13:52:48.355815 3453 raft_consensus.cc:374] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:52:48.355921 3453 raft_consensus.cc:729] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811ee5aef57b441293e37ab5f4847587, State: Initialized, Role: FOLLOWER
I20260927 13:52:48.356019 3304 heartbeater.cc:503] Master 127.20.133.126:37329 was elected leader, sending a full tablet report...
I20260927 13:52:48.356559 3453 consensus_queue.cc:260] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } } peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } }
I20260927 13:52:48.356709 3451 ts_tablet_manager.cc:1428] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 13:52:48.358176 3453 ts_tablet_manager.cc:1428] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260927 13:52:48.381316 3458 raft_consensus.cc:482] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:52:48.381633 3458 raft_consensus.cc:504] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } } peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } }
I20260927 13:52:48.383250 3458 leader_election.cc:288] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 811ee5aef57b441293e37ab5f4847587 (127.20.133.65:45131), 294c2c702597464594e70c7ef56bd642 (127.20.133.67:43781)
I20260927 13:52:48.390725 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.383650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:38911 (local address 127.20.133.67:43781)
0927 13:52:48.383859 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.383865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.384639 (+ 774us) server_negotiation.cc:407] Connection header received
0927 13:52:48.384770 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.384775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.384842 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.384935 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:48.385719 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.386723 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.388861 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.389652 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.389785 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.389840 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.390302 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.390461 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:52:48.390734 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.383662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:43781 (local address 127.20.133.66:38911)
0927 13:52:48.384549 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:48.384565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:48.384662 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:48.385145 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:48.385149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:48.385184 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:48.385559 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.385569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.386898 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.386903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.388691 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.388710 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.389899 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.389906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.390103 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.390141 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 13:52:48.390236 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 13:52:48.390362 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":71,"thread_start_us":359,"threads_started":1}
I20260927 13:52:48.391137 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.383509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:45131 (local address 127.20.133.66:37777)
0927 13:52:48.384392 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:48.384416 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:52:48.384587 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:52:48.384975 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:52:48.384980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:52:48.385008 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:52:48.385457 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.385482 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.386899 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.386903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.388721 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:52:48.388735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.390326 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.390333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:52:48.390545 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.390585 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 13:52:48.390697 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 13:52:48.390831 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":89,"thread_start_us":704,"threads_started":1}
I20260927 13:52:48.391135 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.383871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:37777 (local address 127.20.133.65:45131)
0927 13:52:48.384072 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.384077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.384522 (+ 445us) server_negotiation.cc:407] Connection header received
0927 13:52:48.384760 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.384765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.384842 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.384941 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:52:48.385614 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.386697 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.388890 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.389922 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.390108 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.390173 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.390742 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.390898 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:52:48.392267 3127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21544a9f7e74a9d98b455a0054c11a0" candidate_uuid: "1bd3d760c0ea4719a54d92c3a25100af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811ee5aef57b441293e37ab5f4847587" is_pre_election: true
I20260927 13:52:48.392445 3390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21544a9f7e74a9d98b455a0054c11a0" candidate_uuid: "1bd3d760c0ea4719a54d92c3a25100af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294c2c702597464594e70c7ef56bd642" is_pre_election: true
I20260927 13:52:48.392812 3127 raft_consensus.cc:2454] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd3d760c0ea4719a54d92c3a25100af in term 0.
I20260927 13:52:48.393126 3390 raft_consensus.cc:2454] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd3d760c0ea4719a54d92c3a25100af in term 0.
I20260927 13:52:48.393571 3192 leader_election.cc:302] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd3d760c0ea4719a54d92c3a25100af, 811ee5aef57b441293e37ab5f4847587; no voters:
I20260927 13:52:48.394021 3458 raft_consensus.cc:2781] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:52:48.394169 3458 raft_consensus.cc:482] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:52:48.394299 3458 raft_consensus.cc:3037] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:48.396553 3458 raft_consensus.cc:504] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } } peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } }
I20260927 13:52:48.397280 3458 leader_election.cc:288] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [CANDIDATE]: Term 1 election: Requested vote from peers 811ee5aef57b441293e37ab5f4847587 (127.20.133.65:45131), 294c2c702597464594e70c7ef56bd642 (127.20.133.67:43781)
I20260927 13:52:48.397699 3127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21544a9f7e74a9d98b455a0054c11a0" candidate_uuid: "1bd3d760c0ea4719a54d92c3a25100af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811ee5aef57b441293e37ab5f4847587"
I20260927 13:52:48.397922 3127 raft_consensus.cc:3037] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:48.397917 3390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21544a9f7e74a9d98b455a0054c11a0" candidate_uuid: "1bd3d760c0ea4719a54d92c3a25100af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294c2c702597464594e70c7ef56bd642"
I20260927 13:52:48.398180 3390 raft_consensus.cc:3037] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:52:48.400118 3127 raft_consensus.cc:2454] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd3d760c0ea4719a54d92c3a25100af in term 1.
I20260927 13:52:48.400436 3390 raft_consensus.cc:2454] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd3d760c0ea4719a54d92c3a25100af in term 1.
I20260927 13:52:48.400527 3192 leader_election.cc:302] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd3d760c0ea4719a54d92c3a25100af, 811ee5aef57b441293e37ab5f4847587; no voters:
I20260927 13:52:48.400851 3458 raft_consensus.cc:2781] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:52:48.401279 3458 raft_consensus.cc:686] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 1 LEADER]: Becoming Leader. State: Replica: 1bd3d760c0ea4719a54d92c3a25100af, State: Running, Role: LEADER
I20260927 13:52:48.401804 3458 consensus_queue.cc:237] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } } peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } }
I20260927 13:52:48.406617 2986 catalog_manager.cc:5495] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bd3d760c0ea4719a54d92c3a25100af (127.20.133.66). New cstate: current_term: 1 leader_uuid: "1bd3d760c0ea4719a54d92c3a25100af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bd3d760c0ea4719a54d92c3a25100af" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44463 } health_report { overall_health: HEALTHY } } }
I20260927 13:52:48.434623 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.429167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.20.133.66:44463)
0927 13:52:48.429373 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.429379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.429459 (+ 80us) server_negotiation.cc:407] Connection header received
0927 13:52:48.429701 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.429705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.429782 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.429909 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:52:48.430276 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.430994 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.432439 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.432863 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.432973 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.433483 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:52:48.434007 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:52:48.434142 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.434257 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.434343 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:52:48.442976 3390 raft_consensus.cc:1261] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642 [term 1 FOLLOWER]: Refusing update from remote peer 1bd3d760c0ea4719a54d92c3a25100af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:52:48.442986 3127 raft_consensus.cc:1261] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587 [term 1 FOLLOWER]: Refusing update from remote peer 1bd3d760c0ea4719a54d92c3a25100af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:52:48.443981 3462 consensus_queue.cc:1035] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [LEADER]: Connected to new peer: Peer: permanent_uuid: "294c2c702597464594e70c7ef56bd642" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 43781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:52:48.444371 3458 consensus_queue.cc:1035] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [LEADER]: Connected to new peer: Peer: permanent_uuid: "811ee5aef57b441293e37ab5f4847587" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 45131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 13:52:48.456599 3436 tablet.cc:2369] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:52:48.473896 2986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:52:48.474150 2986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:52:48.477751 2986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Sending DeleteTablet for 3 replicas of tablet b21544a9f7e74a9d98b455a0054c11a0
I20260927 13:52:48.478861 3239 tablet_service.cc:1514] Processing DeleteTablet for tablet b21544a9f7e74a9d98b455a0054c11a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:52:48 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:34252
I20260927 13:52:48.478855 3370 tablet_service.cc:1514] Processing DeleteTablet for tablet b21544a9f7e74a9d98b455a0054c11a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:52:48 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:58674
I20260927 13:52:48.479622 3107 tablet_service.cc:1514] Processing DeleteTablet for tablet b21544a9f7e74a9d98b455a0054c11a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:52:48 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:37172
I20260927 13:52:48.485286 3476 tablet_replica.cc:330] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af: stopping tablet replica
I20260927 13:52:48.485535 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 3024
I20260927 13:52:48.485908 3476 raft_consensus.cc:2229] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:52:48.486685 3476 raft_consensus.cc:2258] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:52:48.487562 3475 tablet_replica.cc:330] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: stopping tablet replica
I20260927 13:52:48.487797 3477 tablet_replica.cc:330] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587: stopping tablet replica
I20260927 13:52:48.488125 3475 raft_consensus.cc:2229] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:52:48.488225 3477 raft_consensus.cc:2229] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:52:48.488767 3475 raft_consensus.cc:2258] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:52:48.488775 3477 raft_consensus.cc:2258] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:52:48.490998 3475 ts_tablet_manager.cc:1905] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:52:48.491024 3477 ts_tablet_manager.cc:1905] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:52:48.498313 3477 ts_tablet_manager.cc:1918] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:52:48.498504 3477 log.cc:1196] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/b21544a9f7e74a9d98b455a0054c11a0
I20260927 13:52:48.498354 3475 ts_tablet_manager.cc:1918] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:52:48.498734 3475 log.cc:1196] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/b21544a9f7e74a9d98b455a0054c11a0
I20260927 13:52:48.499065 3477 ts_tablet_manager.cc:1939] T b21544a9f7e74a9d98b455a0054c11a0 P 811ee5aef57b441293e37ab5f4847587: Deleting consensus metadata
I20260927 13:52:48.499394 3475 ts_tablet_manager.cc:1939] T b21544a9f7e74a9d98b455a0054c11a0 P 294c2c702597464594e70c7ef56bd642: Deleting consensus metadata
I20260927 13:52:48.499747 3476 ts_tablet_manager.cc:1905] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:52:48.501091 2971 catalog_manager.cc:4841] TS 811ee5aef57b441293e37ab5f4847587 (127.20.133.65:45131): tablet b21544a9f7e74a9d98b455a0054c11a0 (table test-table [id=ba8b6667291d4387b031bb051a006d6c]) successfully deleted
I20260927 13:52:48.501353 2974 catalog_manager.cc:4841] TS 294c2c702597464594e70c7ef56bd642 (127.20.133.67:43781): tablet b21544a9f7e74a9d98b455a0054c11a0 (table test-table [id=ba8b6667291d4387b031bb051a006d6c]) successfully deleted
I20260927 13:52:48.508297 3476 ts_tablet_manager.cc:1918] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:52:48.508541 3476 log.cc:1196] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/b21544a9f7e74a9d98b455a0054c11a0
I20260927 13:52:48.508872 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.487272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.20.133.65:45131)
0927 13:52:48.487677 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.487684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.487703 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:52:48.487768 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.487773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.487859 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.487958 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:48.489928 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.490974 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.492931 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.493614 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.493746 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.500902 (+ 7156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:48.500916 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:48.500934 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:48.500980 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:48.505104 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:48.506184 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:48.506196 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:48.506200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:48.506371 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:48.506846 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:48.506852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:48.506855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:48.507366 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:48.507666 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.508038 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.508295 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20260927 13:52:48.509223 3476 ts_tablet_manager.cc:1939] T b21544a9f7e74a9d98b455a0054c11a0 P 1bd3d760c0ea4719a54d92c3a25100af: Deleting consensus metadata
I20260927 13:52:48.509676 3167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:48.511065 2972 catalog_manager.cc:4841] TS 1bd3d760c0ea4719a54d92c3a25100af (127.20.133.66:44463): tablet b21544a9f7e74a9d98b455a0054c11a0 (table test-table [id=ba8b6667291d4387b031bb051a006d6c]) successfully deleted
I20260927 13:52:48.603243 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 3024
I20260927 13:52:48.617981 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 3176
I20260927 13:52:48.632279 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.618772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.20.133.66:44463)
0927 13:52:48.618946 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.618952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.618989 (+ 37us) server_negotiation.cc:407] Connection header received
0927 13:52:48.619188 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.619194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.619263 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.619350 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:48.620764 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.621568 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.622940 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.623430 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.623541 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.626241 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:48.626256 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:48.626274 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:48.626322 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:48.629392 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:48.630148 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:48.630156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:48.630158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:48.630301 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:48.630692 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:48.630696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:48.630698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:48.631086 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:48.631420 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.631717 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.631931 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:52:48.632913 3299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:48.745517 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 3176
I20260927 13:52:48.761061 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 3307
I20260927 13:52:48.775060 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:48.761972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.20.133.67:43781)
0927 13:52:48.762428 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 13:52:48.762434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:52:48.762452 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:52:48.762511 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:52:48.762516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:52:48.762592 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:52:48.762688 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:52:48.764180 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.764973 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.766504 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:52:48.766932 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:52:48.767047 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:52:48.769547 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:52:48.769561 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:52:48.769577 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:52:48.769619 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:52:48.772236 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:48.773007 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:48.773015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:48.773018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:48.773165 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:52:48.773515 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:52:48.773520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:52:48.773522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:52:48.773918 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:52:48.774167 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 13:52:48.774543 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 13:52:48.774731 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
I20260927 13:52:48.775662 3430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:48.878927 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 3307
I20260927 13:52:48.892645 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 2954
I20260927 13:52:48.893911 3017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:52:49.008291 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 2954
I20260927 13:52:49.027181 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:37329
--webserver_interface=127.20.133.126
--webserver_port=46677
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:37329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:52:49.359783 3482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:49.360114 3482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:49.360250 3482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:49.368023 3482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:52:49.368146 3482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:49.368204 3482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:49.368285 3482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:52:49.368367 3482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:52:49.378646 3482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:37329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=46677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:49.380019 3482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:49.381361 3482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:49.393268 3487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:49.393298 3488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:49.393268 3490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:49.394001 3482 server_base.cc:1034] running on GCE node
I20260927 13:52:49.394781 3482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:49.396315 3482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:49.397725 3482 hybrid_clock.cc:648] HybridClock initialized: now 1790517169397631 us; error 39 us; skew 500 ppm
I20260927 13:52:49.398339 3482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:49 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:49.402005 3482 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:52:49.403868 3482 webserver.cc:460] Webserver started at http://127.20.133.126:46677/ using document root <none> and password file <none>
I20260927 13:52:49.404721 3482 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:49.404886 3482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:49.411918 3482 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:52:49.415225 3497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:49.416806 3482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 13:52:49.417003 3482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "0bdddcda75c14811bc5945dea24078bf"
format_stamp: "Formatted at 2026-09-27 13:52:43 on dist-test-slave-q25r"
server_key: "5cb6083ddf77d2160305a37895997cae"
server_key_iv: "a93632c78e597cdfc4c90198b58aece1"
server_key_version: "encryptionkey@0"
I20260927 13:52:49.418085 3482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:49.447083 3482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:49.451496 3482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:49.451802 3482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:49.469791 3482 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:37329
I20260927 13:52:49.469810 3548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:37329 every 8 connection(s)
I20260927 13:52:49.471336 3482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:52:49.479177 3549 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:52:49.480654 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 3482
I20260927 13:52:49.481654 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:45131
--local_ip_for_outbound_sockets=127.20.133.65
--tserver_master_addrs=127.20.133.126:37329
--webserver_port=40981
--webserver_interface=127.20.133.65
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:52:49.483992 3549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Bootstrap starting.
I20260927 13:52:49.506510 3549 log.cc:824] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Log is configured to *not* fsync() on all Append() calls
I20260927 13:52:49.524034 3549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:52:49.524642 3549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf: Bootstrap complete.
I20260927 13:52:49.533831 3549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:49.534814 3549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bdddcda75c14811bc5945dea24078bf, State: Initialized, Role: FOLLOWER
I20260927 13:52:49.535513 3549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:49.535772 3549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:52:49.535885 3549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:52:49.536051 3549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 2 FOLLOWER]: Advancing to term 3
I20260927 13:52:49.540068 3549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:49.540670 3549 leader_election.cc:302] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bdddcda75c14811bc5945dea24078bf; no voters:
I20260927 13:52:49.541321 3549 leader_election.cc:288] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 13:52:49.541474 3553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 3 FOLLOWER]: Leader election won for term 3
I20260927 13:52:49.542534 3553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [term 3 LEADER]: Becoming Leader. State: Replica: 0bdddcda75c14811bc5945dea24078bf, State: Running, Role: LEADER
I20260927 13:52:49.543157 3553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } }
I20260927 13:52:49.543519 3549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:52:49.545737 3555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bdddcda75c14811bc5945dea24078bf. Latest consensus state: current_term: 3 leader_uuid: "0bdddcda75c14811bc5945dea24078bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } } }
I20260927 13:52:49.546191 3555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:49.546595 3554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0bdddcda75c14811bc5945dea24078bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdddcda75c14811bc5945dea24078bf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 37329 } } }
I20260927 13:52:49.546850 3554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdddcda75c14811bc5945dea24078bf [sys.catalog]: This master's current role is: LEADER
I20260927 13:52:49.548676 3561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:52:49.558609 3561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:52:49.560916 3561 catalog_manager.cc:1259] Loaded cluster ID: f47cb84bf3c54b608cd806329fe7f61c
I20260927 13:52:49.561435 3561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 13:52:49.563954 3561 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd1eeeee9ad google::LogMessage::Fail() at ??:0
@ 0x7fd1eeef188a google::LogMessage::SendToLog() at ??:0
@ 0x7fd1eeeee4de google::LogMessage::Flush() at ??:0
@ 0x7fd1eeeefa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd204925512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd204922976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd2049aa610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd2049aa439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd1f1b3a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd1f1b3d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd1f1b3d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd1f1b18eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd1f4df26db start_thread at ??:0
@ 0x7fd1edd8861f clone at ??:0
W20260927 13:52:49.879295 3551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:49.879601 3551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:49.879719 3551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:49.887601 3551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:49.887744 3551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:49.887908 3551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:52:49.898232 3551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:45131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=40981
--tserver_master_addrs=127.20.133.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:49.899569 3551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:49.900955 3551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:49.914242 3590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:49.914137 3593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:49.914162 3591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:49.914987 3551 server_base.cc:1034] running on GCE node
I20260927 13:52:49.915608 3551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:49.916728 3551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:49.918016 3551 hybrid_clock.cc:648] HybridClock initialized: now 1790517169917952 us; error 54 us; skew 500 ppm
I20260927 13:52:49.918401 3551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:49 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:49.926741 3551 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:52:49.928303 3551 webserver.cc:460] Webserver started at http://127.20.133.65:40981/ using document root <none> and password file <none>
I20260927 13:52:49.929051 3551 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:49.929186 3551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:49.934803 3551 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:52:49.937654 3600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:49.939064 3551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:52:49.939245 3551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "811ee5aef57b441293e37ab5f4847587"
format_stamp: "Formatted at 2026-09-27 13:52:44 on dist-test-slave-q25r"
server_key: "2c4e227fadd269977b43383656d0c3c7"
server_key_iv: "b3233d2ac78aea5b0c888a69fc6dde30"
server_key_version: "encryptionkey@0"
I20260927 13:52:49.940255 3551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:49.955004 3551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:49.959455 3551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:49.959739 3551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:49.960785 3551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:49.962162 3551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:49.962257 3551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:49.962365 3551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:49.962435 3551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:49.992748 3551 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:45131
I20260927 13:52:49.992794 3712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:45131 every 8 connection(s)
I20260927 13:52:49.994232 3551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:52:49.999413 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:49.997416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:43275)
0927 13:52:49.998298 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:49.998519 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":371,"threads_started":1}
I20260927 13:52:49.999606 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 3551
W20260927 13:52:50.000173 3713 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:37329 (0 consecutive failures): Network error: Failed to ping master at 127.20.133.126:37329: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
I20260927 13:52:50.000381 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:44463
--local_ip_for_outbound_sockets=127.20.133.66
--tserver_master_addrs=127.20.133.126:37329
--webserver_port=36867
--webserver_interface=127.20.133.66
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:52:50.001206 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:50.000826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:42035)
0927 13:52:50.000986 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:50.001086 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 13:52:50.002122 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:50.001815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:55543)
0927 13:52:50.001929 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:50.002007 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 13:52:50.002357 3713 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 13:52:50.334547 3716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:50.334863 3716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:50.334996 3716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:50.343065 3716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:50.343225 3716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:50.343400 3716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:52:50.354218 3716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:44463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=36867
--tserver_master_addrs=127.20.133.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:50.355566 3716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:50.356870 3716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:50.369623 3721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:50.369638 3724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:50.369856 3722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:50.370507 3716 server_base.cc:1034] running on GCE node
I20260927 13:52:50.371134 3716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:50.372622 3716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:50.373946 3716 hybrid_clock.cc:648] HybridClock initialized: now 1790517170373893 us; error 52 us; skew 500 ppm
I20260927 13:52:50.374285 3716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:50 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517170, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:50.377396 3716 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:52:50.378898 3716 webserver.cc:460] Webserver started at http://127.20.133.66:36867/ using document root <none> and password file <none>
I20260927 13:52:50.379565 3716 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:50.379665 3716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:50.385113 3716 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:52:50.387853 3731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:50.389379 3716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:52:50.389599 3716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "1bd3d760c0ea4719a54d92c3a25100af"
format_stamp: "Formatted at 2026-09-27 13:52:44 on dist-test-slave-q25r"
server_key: "bf3a3b3a340a7d4635210f2f7193275e"
server_key_iv: "268a18db9c47cf4488217dc8a3e3a5a4"
server_key_version: "encryptionkey@0"
I20260927 13:52:50.390605 3716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:50.431516 3716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:50.436576 3716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:50.436910 3716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:50.438082 3716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:50.439625 3716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:50.439740 3716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:50.439869 3716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:50.439961 3716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:50.473049 3716 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:44463
I20260927 13:52:50.473109 3843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:44463 every 8 connection(s)
I20260927 13:52:50.474502 3716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:52:50.479058 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:50.477287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:57365)
0927 13:52:50.478021 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:50.478235 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":287,"threads_started":1}
W20260927 13:52:50.479744 3844 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:37329 (0 consecutive failures): Network error: Failed to ping master at 127.20.133.126:37329: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
I20260927 13:52:50.480544 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:50.480206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:49505)
0927 13:52:50.480370 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:50.480440 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 13:52:50.481302 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:50.481006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:54173)
0927 13:52:50.481133 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:50.481201 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 13:52:50.481516 3844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:52:50.484716 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 3716
I20260927 13:52:50.485282 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:43781
--local_ip_for_outbound_sockets=127.20.133.67
--tserver_master_addrs=127.20.133.126:37329
--webserver_port=39309
--webserver_interface=127.20.133.67
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:52:50.811090 3847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:52:50.811398 3847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:52:50.811520 3847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:52:50.819594 3847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:52:50.819741 3847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:52:50.819897 3847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:52:50.830106 3847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:43781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=39309
--tserver_master_addrs=127.20.133.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:52:50.831424 3847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:52:50.832755 3847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:52:50.846457 3852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:50.846376 3853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:52:50.846426 3855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:52:50.848029 3847 server_base.cc:1034] running on GCE node
I20260927 13:52:50.848799 3847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:52:50.849822 3847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:52:50.851058 3847 hybrid_clock.cc:648] HybridClock initialized: now 1790517170850999 us; error 49 us; skew 500 ppm
I20260927 13:52:50.851399 3847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:52:50 dist-test-slave-q25r krb5kdc[2403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517170, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:52:50.854629 3847 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:52:50.856261 3847 webserver.cc:460] Webserver started at http://127.20.133.67:39309/ using document root <none> and password file <none>
I20260927 13:52:50.857028 3847 fs_manager.cc:359] Metadata directory not provided
I20260927 13:52:50.857132 3847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:52:50.862596 3847 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:52:50.865486 3862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:50.867059 3847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:52:50.867244 3847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "294c2c702597464594e70c7ef56bd642"
format_stamp: "Formatted at 2026-09-27 13:52:45 on dist-test-slave-q25r"
server_key: "297d5b312720e92acac848ae1c71e532"
server_key_iv: "bb389c1f635d7c9115a2adda8cb25172"
server_key_version: "encryptionkey@0"
I20260927 13:52:50.868280 3847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:52:50.906549 3847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:52:50.911401 3847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:52:50.911655 3847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:52:50.912704 3847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:52:50.914160 3847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:52:50.914269 3847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:50.914366 3847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:52:50.914418 3847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:52:50.945544 3847 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:43781
I20260927 13:52:50.945631 3974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:43781 every 8 connection(s)
I20260927 13:52:50.947183 3847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:52:50.948283 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 3847
I20260927 13:52:50.952450 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:50.950452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:40205)
0927 13:52:50.951264 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:50.951496 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":332,"threads_started":1}
W20260927 13:52:50.953403 3975 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:37329 (0 consecutive failures): Network error: Failed to ping master at 127.20.133.126:37329: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
I20260927 13:52:50.954488 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:50.954003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:45141)
0927 13:52:50.954267 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:50.954344 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260927 13:52:50.955523 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:50.955128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:53891)
0927 13:52:50.955311 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:50.955401 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 13:52:50.955783 3975 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:52:51.003906 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:51.002978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:42455)
0927 13:52:51.003528 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:51.003673 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":272,"threads_started":1}
I20260927 13:52:51.482957 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:51.482096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:51769)
0927 13:52:51.482590 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:51.482721 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":236,"threads_started":1}
I20260927 13:52:51.957150 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:51.956241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:41517)
0927 13:52:51.956786 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:51.956913 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":302,"threads_started":1}
I20260927 13:52:52.006174 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:52.005172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:48111)
0927 13:52:52.005852 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:52.005976 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":318,"threads_started":1}
I20260927 13:52:52.485189 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:52.484198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:58047)
0927 13:52:52.484865 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:52.484987 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":344,"threads_started":1}
I20260927 13:52:52.959272 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:52.958301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:39483)
0927 13:52:52.958912 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:52.959039 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":298,"threads_started":1}
I20260927 13:52:53.008523 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:53.007527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:38541)
0927 13:52:53.008130 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:53.008238 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":312,"threads_started":1}
I20260927 13:52:53.487468 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:53.486451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:58755)
0927 13:52:53.487072 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:53.487190 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":324,"threads_started":1}
I20260927 13:52:53.961393 3987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:53.960522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:46905)
0927 13:52:53.961079 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:53.961195 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":266,"threads_started":1}
I20260927 13:52:54.010674 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:54.009701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:56783)
0927 13:52:54.010297 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:54.010424 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":276,"threads_started":1}
I20260927 13:52:54.489464 3989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:54.488652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:51679)
0927 13:52:54.489178 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:54.489274 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":257,"threads_started":1}
I20260927 13:52:54.963186 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:54.962383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:35695)
0927 13:52:54.962886 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:54.962979 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":282,"threads_started":1}
I20260927 13:52:55.012750 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:55.011865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:46561)
0927 13:52:55.012432 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:55.012554 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":281,"threads_started":1}
I20260927 13:52:55.491814 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:55.490841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:38787)
0927 13:52:55.491469 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:55.491583 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":312,"threads_started":1}
I20260927 13:52:55.965716 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:55.964765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:45045)
0927 13:52:55.965372 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:55.965477 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":298,"threads_started":1}
I20260927 13:52:56.014812 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:56.013892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:57489)
0927 13:52:56.014498 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:56.014624 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":326,"threads_started":1}
I20260927 13:52:56.494091 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:56.493119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:55103)
0927 13:52:56.493742 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:56.493880 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":316,"threads_started":1}
I20260927 13:52:56.968035 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:56.966962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:39491)
0927 13:52:56.967664 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:56.967806 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":370,"threads_started":1}
I20260927 13:52:57.017210 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:57.016172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:38367)
0927 13:52:57.016816 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:57.016924 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":327,"threads_started":1}
I20260927 13:52:57.496459 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:57.495316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:53049)
0927 13:52:57.495993 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:57.496116 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":291,"threads_started":1}
I20260927 13:52:57.970374 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:57.969337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:42223)
0927 13:52:57.970005 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:57.970123 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":273,"threads_started":1}
I20260927 13:52:58.019204 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:58.018311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:33691)
0927 13:52:58.018900 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:58.019007 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":297,"threads_started":1}
I20260927 13:52:58.498281 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:58.497513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:42983)
0927 13:52:58.497999 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:58.498097 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
I20260927 13:52:58.972309 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:58.971500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:50267)
0927 13:52:58.972034 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:58.972128 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":265,"threads_started":1}
I20260927 13:52:59.021459 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:59.020405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:43837)
0927 13:52:59.021128 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:59.021244 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":307,"threads_started":1}
I20260927 13:52:59.500424 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:59.499355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:40155)
0927 13:52:59.499960 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:59.500086 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":282,"threads_started":1}
I20260927 13:52:59.974658 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:52:59.973502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:52771)
0927 13:52:59.974210 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 13:52:59.974388 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":316,"threads_started":1}
I20260927 13:53:00.023533 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:00.022569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:41371)
0927 13:53:00.023212 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:00.023320 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":345,"threads_started":1}
I20260927 13:53:00.502601 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:00.501661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:34919)
0927 13:53:00.502267 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:00.502392 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":308,"threads_started":1}
I20260927 13:53:00.976902 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:00.975921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:50619)
0927 13:53:00.976591 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:00.976719 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":342,"threads_started":1}
I20260927 13:53:01.025837 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:01.024814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:60891)
0927 13:53:01.025366 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:01.025524 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":252,"threads_started":1}
I20260927 13:53:01.504889 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:01.503808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:60735)
0927 13:53:01.504486 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:01.504631 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":282,"threads_started":1}
I20260927 13:53:01.979140 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:01.978112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:39245)
0927 13:53:01.978709 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:01.978878 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":297,"threads_started":1}
I20260927 13:53:02.028131 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:02.027181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:56131)
0927 13:53:02.027820 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:02.027940 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":315,"threads_started":1}
I20260927 13:53:02.506804 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:02.505998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:60017)
0927 13:53:02.506516 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:02.506616 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":256,"threads_started":1}
I20260927 13:53:02.981525 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:02.980460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:53433)
0927 13:53:02.981098 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:02.981240 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":323,"threads_started":1}
I20260927 13:53:03.030442 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:03.029361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:43009)
0927 13:53:03.030006 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:03.030126 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":310,"threads_started":1}
I20260927 13:53:03.508939 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:03.507977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:39627)
0927 13:53:03.508619 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:03.508730 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":364,"threads_started":1}
I20260927 13:53:03.983747 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:03.982766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:55593)
0927 13:53:03.983385 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:03.983510 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":293,"threads_started":1}
I20260927 13:53:04.032593 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:04.031600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:38247)
0927 13:53:04.032207 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:04.032309 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":308,"threads_started":1}
I20260927 13:53:04.510814 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:04.509970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:60001)
0927 13:53:04.510491 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:04.510604 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":233,"threads_started":1}
I20260927 13:53:04.985503 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:04.984766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:49809)
0927 13:53:04.985225 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:04.985311 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
I20260927 13:53:05.034667 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:05.033850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.65:39035)
0927 13:53:05.034330 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:05.034435 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":233,"threads_started":1}
I20260927 13:53:05.513093 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:05.512034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.66:58989)
0927 13:53:05.512703 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:05.512854 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":336,"threads_started":1}
I20260927 13:53:05.960222 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 3551
I20260927 13:53:05.983825 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:05.961759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.20.133.65:45131)
0927 13:53:05.962547 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:05.962570 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:05.962629 (+ 59us) server_negotiation.cc:407] Connection header received
0927 13:53:05.964956 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:05.964968 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:05.965562 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:05.965962 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:05.967729 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:05.969304 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:05.970698 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:05.971532 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:05.971741 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:05.975577 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:05.975589 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:05.975616 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:05.975685 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:05.979428 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:05.980525 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:05.980535 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:05.980544 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:05.980778 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:05.981211 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:05.981215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:05.981217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:05.982004 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:05.982363 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:05.982909 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 13:53:05.983485 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":344,"threads_started":1}
I20260927 13:53:05.985647 3708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:53:05.987949 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:05.986952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:37329 (local address 127.20.133.67:59869)
0927 13:53:05.987540 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:05.987695 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":287,"threads_started":1}
I20260927 13:53:06.079386 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 3551
I20260927 13:53:06.094925 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 3716
I20260927 13:53:06.116170 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:06.096194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.20.133.66:44463)
0927 13:53:06.096823 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:06.096844 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:06.096903 (+ 59us) server_negotiation.cc:407] Connection header received
0927 13:53:06.098680 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:06.098692 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:06.099258 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:06.099645 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:06.101469 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:06.102983 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:06.104593 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:06.105481 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:06.105676 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:06.108903 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:06.108912 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:06.108945 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:06.109000 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:06.112444 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:06.113323 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:06.113331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:06.113339 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:06.113539 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:06.113944 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:06.113947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:06.113948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:06.114589 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:06.114861 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:06.115338 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 13:53:06.115838 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":280,"threads_started":1}
I20260927 13:53:06.117871 3839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:53:06.210394 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 3716
I20260927 13:53:06.229395 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 3847
I20260927 13:53:06.253484 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:06.230757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.20.133.67:43781)
0927 13:53:06.231493 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:06.231512 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:06.231564 (+ 52us) server_negotiation.cc:407] Connection header received
0927 13:53:06.233654 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:06.233668 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:06.234345 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:06.234776 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:06.236607 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:06.237983 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:06.239585 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:06.240641 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:06.240854 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:06.244692 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:06.244704 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:06.244724 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:06.244782 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:06.249045 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:06.250291 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:06.250305 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:06.250316 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:06.250657 (+ 341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:06.251090 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:06.251093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:06.251095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:06.251764 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:06.252063 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:06.252682 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 13:53:06.253147 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":334,"threads_started":1}
I20260927 13:53:06.254920 3970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:53:06.345063 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 3847
2026-09-27T13:53:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24717 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:53:06 dist-test-slave-q25r krb5kdc[4033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:53:06 dist-test-slave-q25r krb5kdc[4033](info): set up 2 sockets
Sep 27 13:53:06 dist-test-slave-q25r krb5kdc[4033](info): commencing operation
krb5kdc: starting...
W20260927 13:53:07.368811 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:53:07 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:53:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:53:07.612138 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:43389
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:43389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:53:07.964920 4046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:07.965260 4046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:07.965389 4046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:07.973531 4046 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 13:53:07.973686 4046 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 13:53:07.973729 4046 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 13:53:07.973776 4046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:53:07.973841 4046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:07.973946 4046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:07.973992 4046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:53:07.974069 4046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:53:07.984486 4046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:43389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:43389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:07.985832 4046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:07.987167 4046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:07.999208 4051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:07.999250 4054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:07.999274 4052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:08.000617 4046 server_base.cc:1034] running on GCE node
I20260927 13:53:08.001469 4046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:08.002974 4046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:08.004201 4046 hybrid_clock.cc:648] HybridClock initialized: now 1790517188004135 us; error 45 us; skew 500 ppm
I20260927 13:53:08.004585 4046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:08 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517188, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:08.007722 4046 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:53:08.009323 4046 webserver.cc:460] Webserver started at http://127.20.133.126:45851/ using document root <none> and password file <none>
I20260927 13:53:08.010056 4046 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:08.010160 4046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:08.010459 4046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:53:08.013675 4046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "54c568ad302f48a5badafa05921a6faa"
format_stamp: "Formatted at 2026-09-27 13:53:08 on dist-test-slave-q25r"
server_key: "7da52ce031dce9f652f0f5d074162f16"
server_key_iv: "83a98f2df62a04f8ce8857e39a32a0e9"
server_key_version: "encryptionkey@0"
I20260927 13:53:08.014537 4046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "54c568ad302f48a5badafa05921a6faa"
format_stamp: "Formatted at 2026-09-27 13:53:08 on dist-test-slave-q25r"
server_key: "7da52ce031dce9f652f0f5d074162f16"
server_key_iv: "83a98f2df62a04f8ce8857e39a32a0e9"
server_key_version: "encryptionkey@0"
I20260927 13:53:08.019918 4046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:53:08.024039 4061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:08.025543 4046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 13:53:08.025725 4046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "54c568ad302f48a5badafa05921a6faa"
format_stamp: "Formatted at 2026-09-27 13:53:08 on dist-test-slave-q25r"
server_key: "7da52ce031dce9f652f0f5d074162f16"
server_key_iv: "83a98f2df62a04f8ce8857e39a32a0e9"
server_key_version: "encryptionkey@0"
I20260927 13:53:08.026000 4046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:08.039911 4046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:08.044288 4046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:08.044600 4046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:08.064985 4046 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:43389
I20260927 13:53:08.064993 4112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:43389 every 8 connection(s)
I20260927 13:53:08.066401 4046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:53:08.067823 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 4046
I20260927 13:53:08.068051 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:53:08.069063 21013 external_mini_cluster.cc:1428] Setting key 578f06ca1bf6c3dc78dadffa5e3c053c
I20260927 13:53:08.071396 4113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:53:08 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:53:08.086939 4113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Bootstrap starting.
I20260927 13:53:08.090866 4113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Neither blocks nor log segments found. Creating new log.
I20260927 13:53:08.092265 4113 log.cc:824] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:08.096596 4113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: No bootstrap required, opened a new log
I20260927 13:53:08.097482 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:08.071327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.20.133.126:43389)
0927 13:53:08.072389 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:08.072406 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:08.072851 (+ 445us) server_negotiation.cc:407] Connection header received
0927 13:53:08.074730 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:08.074753 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:08.075188 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:08.075566 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:08.077771 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:08.079250 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:08.081426 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:08.082496 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:08.082681 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:08.088285 (+ 5604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:08.088296 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:08.088312 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:08.088455 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:08.091928 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:08.092997 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:08.093009 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:08.093020 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:08.093301 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:08.094251 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:08.094255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:08.094257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:08.094895 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:08.095148 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:08.095813 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 13:53:08.096216 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":403,"threads_started":1}
I20260927 13:53:08.104857 4113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:08.105299 4113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:53:08.105394 4113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c568ad302f48a5badafa05921a6faa, State: Initialized, Role: FOLLOWER
I20260927 13:53:08.106001 4113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:08.106237 4113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:53:08.106375 4113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:53:08.106541 4113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:53:08.109148 4113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:08.109895 4113 leader_election.cc:302] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54c568ad302f48a5badafa05921a6faa; no voters:
I20260927 13:53:08.110790 4113 leader_election.cc:288] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:53:08.110813 4118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:53:08.112157 4118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 1 LEADER]: Becoming Leader. State: Replica: 54c568ad302f48a5badafa05921a6faa, State: Running, Role: LEADER
I20260927 13:53:08.113039 4118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:08.114199 4113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:53:08.116593 4119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54c568ad302f48a5badafa05921a6faa. Latest consensus state: current_term: 1 leader_uuid: "54c568ad302f48a5badafa05921a6faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } } }
I20260927 13:53:08.117134 4119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: This master's current role is: LEADER
I20260927 13:53:08.116597 4120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54c568ad302f48a5badafa05921a6faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } } }
I20260927 13:53:08.118899 4120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: This master's current role is: LEADER
I20260927 13:53:08.119374 4125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:53:08.125339 4125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:53:08.135280 4125 catalog_manager.cc:1347] Generated new cluster ID: 2e4e32fac308450bb6e2bb190d4d91f9
I20260927 13:53:08.135418 4125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:53:08.153219 4125 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:53:08.154687 4125 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:53:08.162575 4125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Generated new TSK 0
I20260927 13:53:08.163506 4125 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:53:08.225961 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:43389
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:53:08.572939 4139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:08.573292 4139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:08.573457 4139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:08.581121 4139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:08.581260 4139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:08.581420 4139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:53:08.591682 4139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:08.593086 4139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:08.594471 4139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:08.607875 4144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:08.607919 4147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:08.607937 4145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:08.609022 4139 server_base.cc:1034] running on GCE node
I20260927 13:53:08.609692 4139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:08.610740 4139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:08.611981 4139 hybrid_clock.cc:648] HybridClock initialized: now 1790517188611917 us; error 55 us; skew 500 ppm
I20260927 13:53:08.612375 4139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:08 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517188, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:08.615914 4139 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:53:08.617492 4139 webserver.cc:460] Webserver started at http://127.20.133.65:39895/ using document root <none> and password file <none>
I20260927 13:53:08.618194 4139 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:08.618317 4139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:08.618615 4139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:53:08.621651 4139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "c9130609b44d47cc8fa1d0487105b0ff"
format_stamp: "Formatted at 2026-09-27 13:53:08 on dist-test-slave-q25r"
server_key: "a797bc0ecf089d46978b05e4d45cfa9e"
server_key_iv: "c7cd934d6b919066a45a9fead45c3322"
server_key_version: "encryptionkey@0"
I20260927 13:53:08.622370 4139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "c9130609b44d47cc8fa1d0487105b0ff"
format_stamp: "Formatted at 2026-09-27 13:53:08 on dist-test-slave-q25r"
server_key: "a797bc0ecf089d46978b05e4d45cfa9e"
server_key_iv: "c7cd934d6b919066a45a9fead45c3322"
server_key_version: "encryptionkey@0"
I20260927 13:53:08.627497 4139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 13:53:08.631120 4154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:08.632546 4139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:53:08.632728 4139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "c9130609b44d47cc8fa1d0487105b0ff"
format_stamp: "Formatted at 2026-09-27 13:53:08 on dist-test-slave-q25r"
server_key: "a797bc0ecf089d46978b05e4d45cfa9e"
server_key_iv: "c7cd934d6b919066a45a9fead45c3322"
server_key_version: "encryptionkey@0"
I20260927 13:53:08.632911 4139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:08.655056 4139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:08.659868 4139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:08.660189 4139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:08.661234 4139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:53:08.662657 4139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:53:08.662750 4139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:08.662873 4139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:53:08.662925 4139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:08.695982 4139 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:40485
I20260927 13:53:08.696065 4266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:40485 every 8 connection(s)
I20260927 13:53:08.697327 4139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:53:08.706667 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 4139
I20260927 13:53:08.706889 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:53:08.707710 21013 external_mini_cluster.cc:1428] Setting key 8dbd9624e522b76cbda12fcefe76d0b4
Sep 27 13:53:08 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517188, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:53:08.718604 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:08.699813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:35767 (local address 127.20.133.126:43389)
0927 13:53:08.700270 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:08.700277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:08.701201 (+ 924us) server_negotiation.cc:407] Connection header received
0927 13:53:08.702998 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:08.703003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:08.703073 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:08.703169 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:08.704730 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:08.705500 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:08.707109 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:08.707778 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:08.707935 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:08.711646 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:08.711653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:08.711686 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:08.711735 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:08.715014 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:08.715919 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:08.715925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:08.715929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:08.716085 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:08.716621 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:08.716626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:08.716629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:08.716997 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:08.717157 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:08.717944 (+ 787us) server_negotiation.cc:299] Negotiation successful
0927 13:53:08.718192 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":212,"threads_started":1}
I20260927 13:53:08.719522 4270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:08.700311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:35767)
0927 13:53:08.701038 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:08.701074 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:08.702741 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:08.703377 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:08.703385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:08.703902 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:53:08.704551 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:08.704576 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:08.705674 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:08.705678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:08.706938 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:08.706952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:08.708008 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:08.708014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:08.708207 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:08.708914 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:53:08.708947 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:53:08.711389 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:53:08.715204 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:08.715213 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:08.715228 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:08.715784 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:08.716227 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:08.716230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:08.716233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:08.716501 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:08.717152 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:53:08.717159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:53:08.717528 (+ 369us) client_negotiation.cc:769] Sending connection context
0927 13:53:08.717781 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 13:53:08.718206 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":267,"threads_started":1}
I20260927 13:53:08.721316 4267 heartbeater.cc:347] Connected to a master server at 127.20.133.126:43389
I20260927 13:53:08.721655 4267 heartbeater.cc:464] Registering TS with master...
I20260927 13:53:08.722393 4267 heartbeater.cc:511] Master 127.20.133.126:43389 requested a full tablet report, sending...
I20260927 13:53:08.724818 4078 ts_manager.cc:194] Registered new tserver with Master: c9130609b44d47cc8fa1d0487105b0ff (127.20.133.65:40485)
I20260927 13:53:08.726727 4078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:35767
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:53:08.763509 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:43389
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:53:09.093286 4273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:09.093614 4273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:09.093713 4273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:09.101630 4273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:09.101794 4273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:09.101922 4273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:53:09.112007 4273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:09.113410 4273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:09.114720 4273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:09.127952 4278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:09.127916 4281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:09.127898 4279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:09.129094 4273 server_base.cc:1034] running on GCE node
I20260927 13:53:09.129743 4273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:09.130852 4273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:09.132105 4273 hybrid_clock.cc:648] HybridClock initialized: now 1790517189132027 us; error 64 us; skew 500 ppm
I20260927 13:53:09.132562 4273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:09 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517189, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:09.135996 4273 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:53:09.137774 4273 webserver.cc:460] Webserver started at http://127.20.133.66:45447/ using document root <none> and password file <none>
I20260927 13:53:09.138484 4273 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:09.138610 4273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:09.138918 4273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:53:09.142022 4273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "82715806c3854130b0efeb3308034f69"
format_stamp: "Formatted at 2026-09-27 13:53:09 on dist-test-slave-q25r"
server_key: "2e6d18b680044c480dcc63fe0516a3c1"
server_key_iv: "62ccf9efcdb68ee45922c27938220b78"
server_key_version: "encryptionkey@0"
I20260927 13:53:09.142831 4273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "82715806c3854130b0efeb3308034f69"
format_stamp: "Formatted at 2026-09-27 13:53:09 on dist-test-slave-q25r"
server_key: "2e6d18b680044c480dcc63fe0516a3c1"
server_key_iv: "62ccf9efcdb68ee45922c27938220b78"
server_key_version: "encryptionkey@0"
I20260927 13:53:09.148172 4273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 13:53:09.152019 4288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:09.153580 4273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:53:09.153767 4273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "82715806c3854130b0efeb3308034f69"
format_stamp: "Formatted at 2026-09-27 13:53:09 on dist-test-slave-q25r"
server_key: "2e6d18b680044c480dcc63fe0516a3c1"
server_key_iv: "62ccf9efcdb68ee45922c27938220b78"
server_key_version: "encryptionkey@0"
I20260927 13:53:09.153967 4273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:09.164913 4273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:09.169170 4273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:09.169468 4273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:09.170519 4273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:53:09.171981 4273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:53:09.172066 4273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:09.172194 4273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:53:09.172247 4273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:09.203460 4273 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:45449
I20260927 13:53:09.203471 4400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:45449 every 8 connection(s)
I20260927 13:53:09.204782 4273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:53:09.207304 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 4273
I20260927 13:53:09.207505 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:53:09.207974 21013 external_mini_cluster.cc:1428] Setting key 0447329caa2e666227e649d42f3c89eb
Sep 27 13:53:09 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517189, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:53:09.226843 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:09.207464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:58857 (local address 127.20.133.126:43389)
0927 13:53:09.207658 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:09.207665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:09.208898 (+ 1233us) server_negotiation.cc:407] Connection header received
0927 13:53:09.210895 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:09.210901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:09.210985 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:09.211091 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:09.212575 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.213519 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.215373 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.216174 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.216361 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:09.220281 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:09.220288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:09.220292 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:09.220374 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:09.223248 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:09.224174 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:09.224179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:09.224181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:09.224328 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:09.224870 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:09.224874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:09.224876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:09.225193 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:09.225351 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:09.226175 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 13:53:09.226434 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:53:09.227922 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:09.207949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:58857)
0927 13:53:09.208754 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:09.208798 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:09.210598 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:09.211258 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:09.211266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:09.211766 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:53:09.212404 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:09.212424 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.213714 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.213718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:09.215208 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:09.215230 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.216475 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.216482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:09.216722 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:09.217471 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:53:09.217497 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:53:09.219959 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:53:09.223463 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:09.223474 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:09.223490 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:09.224014 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:09.224488 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:09.224494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:09.224499 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:09.224731 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:09.225334 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:53:09.225344 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:53:09.225756 (+ 412us) client_negotiation.cc:769] Sending connection context
0927 13:53:09.226029 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 13:53:09.226627 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":329,"threads_started":1}
I20260927 13:53:09.229614 4401 heartbeater.cc:347] Connected to a master server at 127.20.133.126:43389
I20260927 13:53:09.229986 4401 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
I20260927 13:53:09.230814 4401 heartbeater.cc:511] Master 127.20.133.126:43389 requested a full tablet report, sending...
I20260927 13:53:09.232542 4078 ts_manager.cc:194] Registered new tserver with Master: 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
I20260927 13:53:09.233418 4078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:58857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:53:09.259763 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:43389
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:53:09.582360 4406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:09.582671 4406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:09.582764 4406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:09.590051 4406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:09.590179 4406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:09.590294 4406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:53:09.600090 4406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:09.601452 4406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:09.602713 4406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:09.616150 4411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:09.616150 4412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:09.616215 4414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:09.616880 4406 server_base.cc:1034] running on GCE node
I20260927 13:53:09.617486 4406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:09.618426 4406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:09.619625 4406 hybrid_clock.cc:648] HybridClock initialized: now 1790517189619570 us; error 48 us; skew 500 ppm
I20260927 13:53:09.619928 4406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:09 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517189, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:09.622906 4406 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:53:09.624402 4406 webserver.cc:460] Webserver started at http://127.20.133.67:45427/ using document root <none> and password file <none>
I20260927 13:53:09.625079 4406 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:09.625202 4406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:09.625487 4406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:53:09.628327 4406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "8345f10771e84351a2d97610722b52d4"
format_stamp: "Formatted at 2026-09-27 13:53:09 on dist-test-slave-q25r"
server_key: "3c91f84f4a895465b02e1b945a8de840"
server_key_iv: "6fd32a4eddcd71fe5a15ffe0e4c2bb63"
server_key_version: "encryptionkey@0"
I20260927 13:53:09.629122 4406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "8345f10771e84351a2d97610722b52d4"
format_stamp: "Formatted at 2026-09-27 13:53:09 on dist-test-slave-q25r"
server_key: "3c91f84f4a895465b02e1b945a8de840"
server_key_iv: "6fd32a4eddcd71fe5a15ffe0e4c2bb63"
server_key_version: "encryptionkey@0"
I20260927 13:53:09.634727 4406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:53:09.638335 4421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:09.639681 4406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:53:09.639834 4406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "8345f10771e84351a2d97610722b52d4"
format_stamp: "Formatted at 2026-09-27 13:53:09 on dist-test-slave-q25r"
server_key: "3c91f84f4a895465b02e1b945a8de840"
server_key_iv: "6fd32a4eddcd71fe5a15ffe0e4c2bb63"
server_key_version: "encryptionkey@0"
I20260927 13:53:09.640026 4406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:09.654362 4406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:09.658761 4406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:09.659041 4406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:09.660022 4406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:53:09.661466 4406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:53:09.661552 4406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:09.661677 4406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:53:09.661733 4406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:09.692715 4406 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:36769
I20260927 13:53:09.692759 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:36769 every 8 connection(s)
I20260927 13:53:09.694131 4406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:53:09.702602 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 4406
I20260927 13:53:09.702762 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:53:09.703197 21013 external_mini_cluster.cc:1428] Setting key 16bbd26560a37e4f9a0431be70a7c26a
Sep 27 13:53:09 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517189, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:53:09.717576 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:09.697045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:36669 (local address 127.20.133.126:43389)
0927 13:53:09.697218 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:09.697223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:09.698724 (+ 1501us) server_negotiation.cc:407] Connection header received
0927 13:53:09.700696 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:09.700702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:09.700818 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:09.700912 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:09.702598 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.703503 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.705777 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.706440 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.706676 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:09.710815 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:09.710821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:09.710824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:09.710862 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:09.713739 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:09.714791 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:09.714796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:09.714799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:09.715022 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:09.715609 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:09.715613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:09.715616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:09.715929 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:09.716062 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:09.716912 (+ 850us) server_negotiation.cc:299] Negotiation successful
0927 13:53:09.717135 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:53:09.718636 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:09.697690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:36669)
0927 13:53:09.698540 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:09.698586 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:09.700404 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:09.701166 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:09.701175 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:09.701699 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:53:09.702380 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:09.702408 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.703671 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.703676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:09.705591 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:09.705607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.706968 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.706973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:09.707186 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:09.707898 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:53:09.707924 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:53:09.710534 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:53:09.713953 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:09.713962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:09.713978 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:09.714623 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:09.715187 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:09.715190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:09.715193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:09.715430 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:09.716059 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:53:09.716065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:53:09.716488 (+ 423us) client_negotiation.cc:769] Sending connection context
0927 13:53:09.716737 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 13:53:09.717274 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":363,"threads_started":1}
I20260927 13:53:09.719988 4534 heartbeater.cc:347] Connected to a master server at 127.20.133.126:43389
I20260927 13:53:09.720311 4534 heartbeater.cc:464] Registering TS with master...
I20260927 13:53:09.721081 4534 heartbeater.cc:511] Master 127.20.133.126:43389 requested a full tablet report, sending...
I20260927 13:53:09.722448 4078 ts_manager.cc:194] Registered new tserver with Master: 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769)
I20260927 13:53:09.723271 4078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:36669
I20260927 13:53:09.730134 4267 heartbeater.cc:503] Master 127.20.133.126:43389 was elected leader, sending a full tablet report...
I20260927 13:53:09.732257 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:53:09.753758 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:09.737108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.20.133.126:43389)
0927 13:53:09.737279 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:09.737283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:09.737737 (+ 454us) server_negotiation.cc:407] Connection header received
0927 13:53:09.737888 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:09.737892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:09.737965 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:09.738070 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:09.739627 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.740417 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.741885 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.742346 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.742462 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:09.745039 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:09.745044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:09.745048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:09.745087 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:09.750944 (+ 5857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:09.751834 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:09.751839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:09.751841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:09.751993 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:09.752444 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:09.752448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:09.752450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:09.752824 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:09.753015 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:09.753328 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 13:53:09.753507 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:53:09.758411 4077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:53:09.765898 4077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:53:09.814497 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:09.779149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.20.133.66:45449)
0927 13:53:09.804918 (+ 25769us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:09.804928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:09.806789 (+ 1861us) server_negotiation.cc:407] Connection header received
0927 13:53:09.806945 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:09.806951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:09.807115 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:09.807275 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:53:09.807842 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.809463 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.812010 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.813349 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.813509 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:09.813730 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:09.813961 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 13:53:09.814160 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25578,"thread_start_us":25762,"threads_started":1}
I20260927 13:53:09.814592 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:09.779475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:45449 (local address 127.0.0.1:49282)
0927 13:53:09.806663 (+ 27188us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:09.806683 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:09.806812 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:09.807282 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:09.807287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:09.807310 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:53:09.807687 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:09.807697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.809620 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.809625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:09.811849 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:09.811862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.813581 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.813587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:09.813740 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:09.813783 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 13:53:09.813895 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 13:53:09.814046 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27056,"mutex_wait_us":144,"thread_start_us":24363,"threads_started":1}
I20260927 13:53:09.815531 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:09.778070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40485 (local address 127.0.0.1:55612)
0927 13:53:09.803365 (+ 25295us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:09.803430 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:09.803628 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:09.806588 (+ 2960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:09.806596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:09.806629 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:53:09.806996 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:09.807005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.809859 (+ 2854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.809865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:09.812442 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:09.812452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.814259 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.814265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:09.814441 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:09.814475 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:53:09.814982 (+ 507us) client_negotiation.cc:240] Negotiation successful
0927 13:53:09.815141 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24943,"mutex_wait_us":196,"spinlock_wait_cycles":16128,"thread_start_us":28317,"threads_started":1}
I20260927 13:53:09.815732 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:09.778327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.20.133.65:40485)
0927 13:53:09.805602 (+ 27275us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:09.805611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:09.805641 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:53:09.805745 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:09.805752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:09.806039 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:09.807997 (+ 1958us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:53:09.808222 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.809700 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.812587 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.814012 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.814155 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:09.814364 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:09.815152 (+ 788us) server_negotiation.cc:299] Negotiation successful
0927 13:53:09.815342 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27101,"thread_start_us":26953,"threads_started":1}
I20260927 13:53:09.817180 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:09.778212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:36769 (local address 127.0.0.1:60368)
0927 13:53:09.806002 (+ 27790us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:09.806025 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:09.806159 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:09.807739 (+ 1580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:09.807744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:09.807768 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:53:09.808169 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:09.808182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.809942 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.809954 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:09.812415 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:09.812434 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.816382 (+ 3948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.816389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:09.816558 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:09.816589 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 13:53:09.816693 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 13:53:09.816838 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27651,"mutex_wait_us":111,"thread_start_us":27510,"threads_started":1}
I20260927 13:53:09.818701 4202 tablet_service.cc:1467] Processing CreateTablet for tablet de7799868e354710bb533ca5ac2182b5 (DEFAULT_TABLE table=test-table [id=9c6cead3475c4194aeb5d5419e29439c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:53:09.818702 4336 tablet_service.cc:1467] Processing CreateTablet for tablet de7799868e354710bb533ca5ac2182b5 (DEFAULT_TABLE table=test-table [id=9c6cead3475c4194aeb5d5419e29439c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:53:09.819594 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:09.778609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.20.133.67:36769)
0927 13:53:09.807326 (+ 28717us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:09.807332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:09.807354 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:53:09.807424 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:09.807428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:09.807559 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:09.807713 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:53:09.808385 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.809748 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.814501 (+ 4753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:09.816142 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:09.816282 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:09.816530 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:09.816743 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 13:53:09.819258 (+ 2515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28579,"thread_start_us":27831,"threads_started":1}
I20260927 13:53:09.820196 4202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7799868e354710bb533ca5ac2182b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:53:09.820623 4336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7799868e354710bb533ca5ac2182b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:53:09.823382 4469 tablet_service.cc:1467] Processing CreateTablet for tablet de7799868e354710bb533ca5ac2182b5 (DEFAULT_TABLE table=test-table [id=9c6cead3475c4194aeb5d5419e29439c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:53:09.824874 4469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7799868e354710bb533ca5ac2182b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:53:09.831693 4552 tablet_bootstrap.cc:492] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: Bootstrap starting.
I20260927 13:53:09.834830 4552 tablet_bootstrap.cc:654] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: Neither blocks nor log segments found. Creating new log.
I20260927 13:53:09.835819 4552 log.cc:824] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:09.837190 4553 tablet_bootstrap.cc:492] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: Bootstrap starting.
I20260927 13:53:09.838909 4555 tablet_bootstrap.cc:492] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: Bootstrap starting.
I20260927 13:53:09.841040 4553 tablet_bootstrap.cc:654] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: Neither blocks nor log segments found. Creating new log.
I20260927 13:53:09.841647 4555 tablet_bootstrap.cc:654] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: Neither blocks nor log segments found. Creating new log.
I20260927 13:53:09.842278 4553 log.cc:824] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:09.842517 4555 log.cc:824] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:09.843122 4552 tablet_bootstrap.cc:492] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: No bootstrap required, opened a new log
I20260927 13:53:09.843483 4552 ts_tablet_manager.cc:1397] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260927 13:53:09.848245 4553 tablet_bootstrap.cc:492] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: No bootstrap required, opened a new log
I20260927 13:53:09.848464 4555 tablet_bootstrap.cc:492] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: No bootstrap required, opened a new log
I20260927 13:53:09.848695 4553 ts_tablet_manager.cc:1397] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20260927 13:53:09.848706 4555 ts_tablet_manager.cc:1397] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260927 13:53:09.854051 4552 raft_consensus.cc:348] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:09.854627 4552 raft_consensus.cc:374] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:53:09.854736 4552 raft_consensus.cc:729] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82715806c3854130b0efeb3308034f69, State: Initialized, Role: FOLLOWER
I20260927 13:53:09.855461 4552 consensus_queue.cc:260] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:09.857133 4401 heartbeater.cc:503] Master 127.20.133.126:43389 was elected leader, sending a full tablet report...
I20260927 13:53:09.857388 4553 raft_consensus.cc:348] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:09.857841 4553 raft_consensus.cc:374] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:53:09.857860 4552 ts_tablet_manager.cc:1428] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 13:53:09.858034 4553 raft_consensus.cc:729] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9130609b44d47cc8fa1d0487105b0ff, State: Initialized, Role: FOLLOWER
I20260927 13:53:09.858671 4553 consensus_queue.cc:260] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:09.859267 4555 raft_consensus.cc:348] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:09.859730 4555 raft_consensus.cc:374] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:53:09.859822 4555 raft_consensus.cc:729] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8345f10771e84351a2d97610722b52d4, State: Initialized, Role: FOLLOWER
I20260927 13:53:09.860800 4555 consensus_queue.cc:260] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:09.863583 4553 ts_tablet_manager.cc:1428] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260927 13:53:09.865171 4534 heartbeater.cc:503] Master 127.20.133.126:43389 was elected leader, sending a full tablet report...
I20260927 13:53:09.865911 4555 ts_tablet_manager.cc:1428] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
W20260927 13:53:09.945945 4535 tablet.cc:2369] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:53:09.949744 4268 tablet.cc:2369] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:53:09.956969 4402 tablet.cc:2369] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:53:10.022511 4558 raft_consensus.cc:482] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:53:10.022830 4558 raft_consensus.cc:504] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:10.024683 4558 leader_election.cc:288] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9130609b44d47cc8fa1d0487105b0ff (127.20.133.65:40485), 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769)
I20260927 13:53:10.031883 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:10.025232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:42701 (local address 127.20.133.67:36769)
0927 13:53:10.025446 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:10.025451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:10.026268 (+ 817us) server_negotiation.cc:407] Connection header received
0927 13:53:10.026420 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:10.026423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:10.026482 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:10.026561 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:53:10.027217 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.027951 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.030014 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.030908 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.031072 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:10.031153 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:10.031501 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 13:53:10.031638 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 13:53:10.031888 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:10.025218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:58553 (local address 127.20.133.65:40485)
0927 13:53:10.025465 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:10.025471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:10.026198 (+ 727us) server_negotiation.cc:407] Connection header received
0927 13:53:10.026380 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:10.026385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:10.026467 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:10.026538 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:53:10.027132 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.027869 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.030065 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.030978 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.031140 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:10.031213 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:10.031500 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 13:53:10.031632 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 13:53:10.032122 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:10.025237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:36769 (local address 127.20.133.66:42701)
0927 13:53:10.026143 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:10.026160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:10.026292 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:10.026590 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:10.026594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:10.026684 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:53:10.027061 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:10.027073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.028084 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.028088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:10.029859 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:10.029876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.031156 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.031171 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:10.031334 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:10.031369 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:53:10.031455 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 13:53:10.031606 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":25,"spinlock_wait_cycles":28416,"thread_start_us":539,"threads_started":1}
I20260927 13:53:10.032146 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:10.025200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40485 (local address 127.20.133.66:58553)
0927 13:53:10.026074 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:10.026098 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:10.026235 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:10.026579 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:10.026584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:10.026608 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:53:10.026991 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:10.027004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.027997 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.028002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:10.029887 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:10.029899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.031216 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.031221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:10.031345 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:10.031372 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 13:53:10.031455 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 13:53:10.031611 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":121,"spinlock_wait_cycles":32512,"thread_start_us":520,"threads_started":1}
I20260927 13:53:10.033427 4222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7799868e354710bb533ca5ac2182b5" candidate_uuid: "82715806c3854130b0efeb3308034f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9130609b44d47cc8fa1d0487105b0ff" is_pre_election: true
I20260927 13:53:10.033417 4489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7799868e354710bb533ca5ac2182b5" candidate_uuid: "82715806c3854130b0efeb3308034f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8345f10771e84351a2d97610722b52d4" is_pre_election: true
I20260927 13:53:10.034018 4489 raft_consensus.cc:2454] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82715806c3854130b0efeb3308034f69 in term 0.
I20260927 13:53:10.034018 4222 raft_consensus.cc:2454] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82715806c3854130b0efeb3308034f69 in term 0.
I20260927 13:53:10.034765 4289 leader_election.cc:302] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 82715806c3854130b0efeb3308034f69, 8345f10771e84351a2d97610722b52d4, c9130609b44d47cc8fa1d0487105b0ff; no voters:
I20260927 13:53:10.035226 4558 raft_consensus.cc:2781] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:53:10.035385 4558 raft_consensus.cc:482] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:53:10.035482 4558 raft_consensus.cc:3037] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:53:10.037909 4558 raft_consensus.cc:504] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:10.038509 4558 leader_election.cc:288] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [CANDIDATE]: Term 1 election: Requested vote from peers c9130609b44d47cc8fa1d0487105b0ff (127.20.133.65:40485), 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769)
I20260927 13:53:10.039075 4222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7799868e354710bb533ca5ac2182b5" candidate_uuid: "82715806c3854130b0efeb3308034f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9130609b44d47cc8fa1d0487105b0ff"
I20260927 13:53:10.039278 4222 raft_consensus.cc:3037] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:53:10.040681 4489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7799868e354710bb533ca5ac2182b5" candidate_uuid: "82715806c3854130b0efeb3308034f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8345f10771e84351a2d97610722b52d4"
I20260927 13:53:10.040910 4489 raft_consensus.cc:3037] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:53:10.042630 4222 raft_consensus.cc:2454] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82715806c3854130b0efeb3308034f69 in term 1.
I20260927 13:53:10.043426 4489 raft_consensus.cc:2454] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82715806c3854130b0efeb3308034f69 in term 1.
I20260927 13:53:10.043439 4290 leader_election.cc:302] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82715806c3854130b0efeb3308034f69, c9130609b44d47cc8fa1d0487105b0ff; no voters:
I20260927 13:53:10.043949 4558 raft_consensus.cc:2781] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:53:10.044576 4558 raft_consensus.cc:686] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 1 LEADER]: Becoming Leader. State: Replica: 82715806c3854130b0efeb3308034f69, State: Running, Role: LEADER
I20260927 13:53:10.045171 4558 consensus_queue.cc:237] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:10.049952 4077 catalog_manager.cc:5495] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82715806c3854130b0efeb3308034f69 (127.20.133.66). New cstate: current_term: 1 leader_uuid: "82715806c3854130b0efeb3308034f69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } health_report { overall_health: HEALTHY } } }
I20260927 13:53:10.088150 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:10.082894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.20.133.66:45449)
0927 13:53:10.083108 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:10.083114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:10.083135 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:53:10.083261 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:10.083265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:10.083356 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:10.083494 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:53:10.083865 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.084615 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.086014 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.086419 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.086534 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:10.087017 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:53:10.087542 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:53:10.087683 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:10.087779 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 13:53:10.087868 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:53:10.096805 4222 raft_consensus.cc:1261] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Refusing update from remote peer 82715806c3854130b0efeb3308034f69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:53:10.096807 4489 raft_consensus.cc:1261] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4 [term 1 FOLLOWER]: Refusing update from remote peer 82715806c3854130b0efeb3308034f69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:53:10.097899 4563 consensus_queue.cc:1035] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:53:10.098255 4558 consensus_queue.cc:1035] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:53:10.125242 4078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:53:10.125509 4078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:53:10.129288 4078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Sending DeleteTablet for 3 replicas of tablet de7799868e354710bb533ca5ac2182b5
I20260927 13:53:10.130496 4469 tablet_service.cc:1514] Processing DeleteTablet for tablet de7799868e354710bb533ca5ac2182b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:53:10 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:60368
I20260927 13:53:10.130614 4202 tablet_service.cc:1514] Processing DeleteTablet for tablet de7799868e354710bb533ca5ac2182b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:53:10 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:55612
I20260927 13:53:10.130978 4336 tablet_service.cc:1514] Processing DeleteTablet for tablet de7799868e354710bb533ca5ac2182b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:53:10 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:49282
I20260927 13:53:10.132318 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 4139
I20260927 13:53:10.142007 4577 tablet_replica.cc:330] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: stopping tablet replica
I20260927 13:53:10.142204 4575 tablet_replica.cc:330] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: stopping tablet replica
I20260927 13:53:10.142621 4575 raft_consensus.cc:2229] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:53:10.142715 4577 raft_consensus.cc:2229] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:53:10.143150 4575 raft_consensus.cc:2258] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:53:10.143543 4577 raft_consensus.cc:2258] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:53:10.145952 4575 ts_tablet_manager.cc:1905] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:53:10.146826 4576 tablet_replica.cc:330] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: stopping tablet replica
I20260927 13:53:10.147302 4576 raft_consensus.cc:2229] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:53:10.147850 4576 raft_consensus.cc:2258] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:53:10.150354 4577 ts_tablet_manager.cc:1905] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:53:10.154148 4575 ts_tablet_manager.cc:1918] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:53:10.154438 4575 log.cc:1196] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/de7799868e354710bb533ca5ac2182b5
Sep 27 13:53:10 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:53:10.155112 4575 ts_tablet_manager.cc:1939] T de7799868e354710bb533ca5ac2182b5 P 8345f10771e84351a2d97610722b52d4: Deleting consensus metadata
I20260927 13:53:10.156316 4576 ts_tablet_manager.cc:1905] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:53:10.157342 4062 catalog_manager.cc:4841] TS 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769): tablet de7799868e354710bb533ca5ac2182b5 (table test-table [id=9c6cead3475c4194aeb5d5419e29439c]) successfully deleted
I20260927 13:53:10.157738 4577 ts_tablet_manager.cc:1918] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:53:10.157912 4577 log.cc:1196] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/de7799868e354710bb533ca5ac2182b5
I20260927 13:53:10.158433 4577 ts_tablet_manager.cc:1939] T de7799868e354710bb533ca5ac2182b5 P 82715806c3854130b0efeb3308034f69: Deleting consensus metadata
I20260927 13:53:10.160157 4065 catalog_manager.cc:4841] TS 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449): tablet de7799868e354710bb533ca5ac2182b5 (table test-table [id=9c6cead3475c4194aeb5d5419e29439c]) successfully deleted
I20260927 13:53:10.164551 4576 ts_tablet_manager.cc:1918] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:53:10.164762 4576 log.cc:1196] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/de7799868e354710bb533ca5ac2182b5
I20260927 13:53:10.165208 4576 ts_tablet_manager.cc:1939] T de7799868e354710bb533ca5ac2182b5 P c9130609b44d47cc8fa1d0487105b0ff: Deleting consensus metadata
I20260927 13:53:10.167194 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:10.133901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.20.133.65:40485)
0927 13:53:10.134173 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:10.134179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:10.138191 (+ 4012us) server_negotiation.cc:407] Connection header received
0927 13:53:10.143718 (+ 5527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:10.143725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:10.143843 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:10.143937 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:10.146242 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.147506 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.149541 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.150135 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.150240 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:10.159034 (+ 8794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:10.159049 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:10.159070 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:10.159118 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:10.163327 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:10.164459 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:10.164469 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:10.164473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:10.164728 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:10.165278 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:10.165283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:10.165286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:10.165813 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:10.166120 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:10.166569 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 13:53:10.166827 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 13:53:10.167214 4063 catalog_manager.cc:4841] TS c9130609b44d47cc8fa1d0487105b0ff (127.20.133.65:40485): tablet de7799868e354710bb533ca5ac2182b5 (table test-table [id=9c6cead3475c4194aeb5d5419e29439c]) successfully deleted
I20260927 13:53:10.167927 4262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:53:10.280182 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 4139
I20260927 13:53:10.295083 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 4273
Sep 27 13:53:10 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:53:10.313045 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:10.295897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.20.133.66:45449)
0927 13:53:10.296080 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:10.296085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:10.296144 (+ 59us) server_negotiation.cc:407] Connection header received
0927 13:53:10.296327 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:10.296359 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:10.296458 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:10.296548 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:10.298116 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.299028 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.300360 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.300793 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.300902 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:10.305317 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:10.305335 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:10.305362 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:10.305413 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:10.309338 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:10.310186 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:10.310196 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:10.310200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:10.310385 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:10.310856 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:10.310861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:10.310865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:10.311357 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:10.311841 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:10.312320 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 13:53:10.312633 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:53:10.313872 4396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:53:10.431105 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 4273
I20260927 13:53:10.446251 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 4406
Sep 27 13:53:10 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:53:10.461738 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:10.446980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.20.133.67:36769)
0927 13:53:10.447125 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:10.447130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:10.447184 (+ 54us) server_negotiation.cc:407] Connection header received
0927 13:53:10.447342 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:10.447346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:10.447410 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:10.447484 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:10.448775 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.449473 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.450858 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:10.451239 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:10.451331 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:10.455917 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:10.455937 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:10.455953 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:10.456002 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:10.458794 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:10.459544 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:10.459550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:10.459553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:10.459715 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:10.460099 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:10.460104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:10.460106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:10.460668 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:10.460897 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:10.461246 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 13:53:10.461418 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 13:53:10.462337 4529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:53:10.551322 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 4406
I20260927 13:53:10.565069 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 4046
I20260927 13:53:10.566205 4108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:53:10.679299 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 4046
I20260927 13:53:30.697968 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:43389
--webserver_interface=127.20.133.126
--webserver_port=45851
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:43389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:53:31.074088 4583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:31.074431 4583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:31.074584 4583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:31.082188 4583 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 13:53:31.082317 4583 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 13:53:31.082373 4583 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 13:53:31.082446 4583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:53:31.082522 4583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:31.082628 4583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:31.082693 4583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:53:31.082770 4583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:53:31.093487 4583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:43389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:43389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=45851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:31.094900 4583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:31.096293 4583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:31.122383 4589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:31.122385 4592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:31.122649 4590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:31.123660 4583 server_base.cc:1034] running on GCE node
I20260927 13:53:31.124788 4583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:31.126629 4583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:31.127931 4583 hybrid_clock.cc:648] HybridClock initialized: now 1790517211127848 us; error 64 us; skew 500 ppm
I20260927 13:53:31.128461 4583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:31 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517211, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:31.138273 4583 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:53:31.140296 4583 webserver.cc:460] Webserver started at http://127.20.133.126:45851/ using document root <none> and password file <none>
I20260927 13:53:31.141192 4583 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:31.141319 4583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:31.148036 4583 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:53:31.151651 4599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:31.153558 4583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:53:31.153797 4583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "54c568ad302f48a5badafa05921a6faa"
format_stamp: "Formatted at 2026-09-27 13:53:08 on dist-test-slave-q25r"
server_key: "7da52ce031dce9f652f0f5d074162f16"
server_key_iv: "83a98f2df62a04f8ce8857e39a32a0e9"
server_key_version: "encryptionkey@0"
I20260927 13:53:31.155001 4583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:31.181928 4583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:31.186972 4583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:31.187294 4583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:31.208729 4583 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:43389
I20260927 13:53:31.208791 4650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:43389 every 8 connection(s)
I20260927 13:53:31.210641 4583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:53:31.218158 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 4583
I20260927 13:53:31.219309 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:40485
--local_ip_for_outbound_sockets=127.20.133.65
--tserver_master_addrs=127.20.133.126:43389
--webserver_port=39895
--webserver_interface=127.20.133.65
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:53:31.220932 4651 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:53:31.226444 4651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Bootstrap starting.
I20260927 13:53:31.253050 4651 log.cc:824] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:31.267666 4651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:53:31.268308 4651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Bootstrap complete.
I20260927 13:53:31.279143 4651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:31.280200 4651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c568ad302f48a5badafa05921a6faa, State: Initialized, Role: FOLLOWER
I20260927 13:53:31.281000 4651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:31.281294 4651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:53:31.281421 4651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:53:31.281702 4651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:53:31.286520 4651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:31.287222 4651 leader_election.cc:302] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54c568ad302f48a5badafa05921a6faa; no voters:
I20260927 13:53:31.287954 4651 leader_election.cc:288] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:53:31.288113 4655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:53:31.289595 4655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 2 LEADER]: Becoming Leader. State: Replica: 54c568ad302f48a5badafa05921a6faa, State: Running, Role: LEADER
I20260927 13:53:31.290294 4655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:31.290946 4651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:53:31.293596 4656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "54c568ad302f48a5badafa05921a6faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } } }
I20260927 13:53:31.294098 4656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: This master's current role is: LEADER
I20260927 13:53:31.294775 4657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54c568ad302f48a5badafa05921a6faa. Latest consensus state: current_term: 2 leader_uuid: "54c568ad302f48a5badafa05921a6faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } } }
I20260927 13:53:31.295118 4657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: This master's current role is: LEADER
I20260927 13:53:31.305531 4661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:53:31.316505 4661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:53:31.318532 4661 catalog_manager.cc:1259] Loaded cluster ID: 2e4e32fac308450bb6e2bb190d4d91f9
I20260927 13:53:31.318691 4661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:53:31.323944 4661 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:53:31.326146 4661 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Loaded TSK: 0
I20260927 13:53:31.353880 4661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Generated new TSK 1
I20260927 13:53:31.354384 4661 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 13:53:31.614467 4653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:31.614776 4653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:31.614883 4653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:31.622443 4653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:31.622575 4653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:31.622720 4653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:53:31.632902 4653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:40485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=39895
--tserver_master_addrs=127.20.133.126:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:31.634307 4653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:31.635679 4653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:31.650561 4678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:31.650242 4679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:31.652029 4681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:31.653465 4653 server_base.cc:1034] running on GCE node
I20260927 13:53:31.654325 4653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:31.655512 4653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:31.657003 4653 hybrid_clock.cc:648] HybridClock initialized: now 1790517211656912 us; error 80 us; skew 500 ppm
I20260927 13:53:31.657392 4653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:31 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517211, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:31.661072 4653 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:53:31.662791 4653 webserver.cc:460] Webserver started at http://127.20.133.65:39895/ using document root <none> and password file <none>
I20260927 13:53:31.663535 4653 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:31.663673 4653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:31.669273 4653 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:53:31.672045 4688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:31.673516 4653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:53:31.673728 4653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "c9130609b44d47cc8fa1d0487105b0ff"
format_stamp: "Formatted at 2026-09-27 13:53:08 on dist-test-slave-q25r"
server_key: "a797bc0ecf089d46978b05e4d45cfa9e"
server_key_iv: "c7cd934d6b919066a45a9fead45c3322"
server_key_version: "encryptionkey@0"
I20260927 13:53:31.675091 4653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:31.710958 4653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:31.715464 4653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:31.715737 4653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:31.716869 4653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:53:31.718312 4653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:53:31.718411 4653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:31.718518 4653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:53:31.718588 4653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:31.754626 4653 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:40485
I20260927 13:53:31.754725 4800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:40485 every 8 connection(s)
I20260927 13:53:31.756156 4653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:53:31.765786 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 4653
I20260927 13:53:31.766718 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:45449
--local_ip_for_outbound_sockets=127.20.133.66
--tserver_master_addrs=127.20.133.126:43389
--webserver_port=45447
--webserver_interface=127.20.133.66
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:53:31 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517211, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:53:31.786842 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:31.759867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:37193)
0927 13:53:31.760826 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:31.760880 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:31.762974 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:31.764613 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:31.764624 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:31.765514 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:53:31.766388 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:31.766417 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:31.768869 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:31.768875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:31.770855 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:31.770872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:31.772230 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:31.772237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:31.772595 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:31.773508 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:53:31.773543 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:53:31.776728 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:53:31.781601 (+ 4873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:31.781614 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:31.781638 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:31.782195 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:31.782768 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:31.782771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:31.782774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:31.782970 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:31.784129 (+ 1159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:53:31.784136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:53:31.784619 (+ 483us) client_negotiation.cc:769] Sending connection context
0927 13:53:31.784914 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 13:53:31.785458 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":399,"threads_started":1}
I20260927 13:53:31.787189 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:31.759594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:37193 (local address 127.20.133.126:43389)
0927 13:53:31.760315 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:31.760371 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:31.761058 (+ 687us) server_negotiation.cc:407] Connection header received
0927 13:53:31.763623 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:31.763636 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:31.764035 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:31.764427 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:31.766826 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:31.768602 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:31.771081 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:31.771901 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:31.772098 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:31.777225 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:31.777236 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:31.777257 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:31.777313 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:31.781378 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:31.782389 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:31.782400 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:31.782410 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:31.782658 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:31.783094 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:31.783098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:31.783101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:31.783791 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:31.784211 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:31.785228 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0927 13:53:31.785794 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":301,"threads_started":1}
I20260927 13:53:31.790787 4801 heartbeater.cc:347] Connected to a master server at 127.20.133.126:43389
I20260927 13:53:31.791244 4801 heartbeater.cc:464] Registering TS with master...
I20260927 13:53:31.792325 4801 heartbeater.cc:511] Master 127.20.133.126:43389 requested a full tablet report, sending...
I20260927 13:53:31.795491 4616 ts_manager.cc:194] Registered new tserver with Master: c9130609b44d47cc8fa1d0487105b0ff (127.20.133.65:40485)
I20260927 13:53:31.797526 4616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:37193
W20260927 13:53:32.115244 4805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:32.115564 4805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:32.115705 4805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:32.123482 4805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:32.123658 4805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:32.123811 4805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:53:32.134435 4805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:45449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=45447
--tserver_master_addrs=127.20.133.126:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:32.135818 4805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:32.137147 4805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:32.150252 4813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:32.150506 4810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:32.150549 4811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:32.152051 4805 server_base.cc:1034] running on GCE node
I20260927 13:53:32.152844 4805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:32.153916 4805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:32.155167 4805 hybrid_clock.cc:648] HybridClock initialized: now 1790517212155110 us; error 41 us; skew 500 ppm
I20260927 13:53:32.155550 4805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:32 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517212, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:32.158922 4805 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:53:32.160604 4805 webserver.cc:460] Webserver started at http://127.20.133.66:45447/ using document root <none> and password file <none>
I20260927 13:53:32.161319 4805 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:32.161448 4805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:32.167079 4805 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 13:53:32.169812 4820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:32.171375 4805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:53:32.171542 4805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "82715806c3854130b0efeb3308034f69"
format_stamp: "Formatted at 2026-09-27 13:53:09 on dist-test-slave-q25r"
server_key: "2e6d18b680044c480dcc63fe0516a3c1"
server_key_iv: "62ccf9efcdb68ee45922c27938220b78"
server_key_version: "encryptionkey@0"
I20260927 13:53:32.172600 4805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:32.188265 4805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:32.192720 4805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:32.192983 4805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:32.193992 4805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:53:32.195720 4805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:53:32.195813 4805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:32.195919 4805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:53:32.195987 4805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:32.232728 4805 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:45449
I20260927 13:53:32.232787 4932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:45449 every 8 connection(s)
I20260927 13:53:32.234278 4805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:53:32.240394 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 4805
I20260927 13:53:32.241236 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:36769
--local_ip_for_outbound_sockets=127.20.133.67
--tserver_master_addrs=127.20.133.126:43389
--webserver_port=45427
--webserver_interface=127.20.133.67
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:53:32 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517212, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:53:32.258091 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.237011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:58291 (local address 127.20.133.126:43389)
0927 13:53:32.237202 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:32.237206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:32.238385 (+ 1179us) server_negotiation.cc:407] Connection header received
0927 13:53:32.240412 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:32.240419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:32.240571 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:32.240694 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:32.243104 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.244488 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.246808 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.247359 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.247487 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.251922 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:32.251928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:32.251931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:32.251971 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:32.254504 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:32.255546 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:32.255550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:32.255553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:32.255680 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:32.256159 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:32.256163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:32.256165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:32.256543 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:32.256652 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:32.257584 (+ 932us) server_negotiation.cc:299] Negotiation successful
0927 13:53:32.257781 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:53:32.259351 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.237462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:58291)
0927 13:53:32.238219 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:32.238256 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:32.240015 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:32.241063 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:32.241075 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:32.241947 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:53:32.242837 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.242865 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.244719 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.244724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.246574 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.246591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.247621 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.247630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.247894 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.248813 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:53:32.248844 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:53:32.251606 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:53:32.254704 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:32.254715 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:32.254736 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:32.255389 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:32.255791 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:32.255795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:32.255799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:32.256039 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:32.256674 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:53:32.256682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:53:32.257126 (+ 444us) client_negotiation.cc:769] Sending connection context
0927 13:53:32.257486 (+ 360us) client_negotiation.cc:240] Negotiation successful
0927 13:53:32.258014 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":297,"threads_started":1}
I20260927 13:53:32.260833 4933 heartbeater.cc:347] Connected to a master server at 127.20.133.126:43389
I20260927 13:53:32.261134 4933 heartbeater.cc:464] Registering TS with master...
I20260927 13:53:32.261853 4933 heartbeater.cc:511] Master 127.20.133.126:43389 requested a full tablet report, sending...
I20260927 13:53:32.263381 4616 ts_manager.cc:194] Registered new tserver with Master: 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449)
I20260927 13:53:32.264212 4616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:58291
W20260927 13:53:32.583410 4936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:32.583727 4936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:32.583858 4936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:32.591620 4936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:32.591753 4936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:32.591876 4936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:53:32.602576 4936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:36769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=45427
--tserver_master_addrs=127.20.133.126:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:32.604017 4936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:32.605455 4936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:32.619179 4944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:32.619243 4942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:32.619262 4941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:32.620039 4936 server_base.cc:1034] running on GCE node
I20260927 13:53:32.620754 4936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:32.621801 4936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:32.623031 4936 hybrid_clock.cc:648] HybridClock initialized: now 1790517212622978 us; error 46 us; skew 500 ppm
I20260927 13:53:32.623396 4936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:32 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517212, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:32.626729 4936 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:53:32.628314 4936 webserver.cc:460] Webserver started at http://127.20.133.67:45427/ using document root <none> and password file <none>
I20260927 13:53:32.629065 4936 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:32.629196 4936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:32.634920 4936 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260927 13:53:32.638099 4951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:32.639652 4936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:53:32.639837 4936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "8345f10771e84351a2d97610722b52d4"
format_stamp: "Formatted at 2026-09-27 13:53:09 on dist-test-slave-q25r"
server_key: "3c91f84f4a895465b02e1b945a8de840"
server_key_iv: "6fd32a4eddcd71fe5a15ffe0e4c2bb63"
server_key_version: "encryptionkey@0"
I20260927 13:53:32.640919 4936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:32.672144 4936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:32.677043 4936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:32.677337 4936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:32.678372 4936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:53:32.679898 4936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:53:32.679991 4936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:32.680088 4936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:53:32.680156 4936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:32.712217 4936 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:36769
I20260927 13:53:32.712268 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:36769 every 8 connection(s)
I20260927 13:53:32.713708 4936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:53:32.721176 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 4936
Sep 27 13:53:32 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517212, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:53:32.734881 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.716184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:46757 (local address 127.20.133.126:43389)
0927 13:53:32.716397 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:32.716402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:32.717540 (+ 1138us) server_negotiation.cc:407] Connection header received
0927 13:53:32.719350 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:32.719355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:32.719437 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:32.719533 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:32.721090 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.721853 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.724041 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.724527 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.724626 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.728649 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:32.728654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:32.728657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:32.728686 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:32.731498 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:32.732280 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:32.732285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:32.732289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:32.732451 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:32.732875 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:32.732880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:32.732883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:32.733260 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:32.733440 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:32.734308 (+ 868us) server_negotiation.cc:299] Negotiation successful
0927 13:53:32.734550 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:53:32.736450 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.716692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:46757)
0927 13:53:32.717381 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:32.717417 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:32.719079 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:32.719773 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:32.719781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:32.720314 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:53:32.720920 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.720937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.722043 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.722047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.723886 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.723903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.724760 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.724768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.725043 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.725718 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:53:32.725741 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:53:32.728425 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:53:32.731638 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:32.731648 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:32.731669 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:32.732143 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:32.732561 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:32.732565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:32.732567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:32.732755 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:32.733427 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:53:32.733442 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:53:32.733815 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 13:53:32.734229 (+ 414us) client_negotiation.cc:240] Negotiation successful
0927 13:53:32.734798 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":261,"threads_started":1}
I20260927 13:53:32.736696 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.722069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.20.133.126:43389)
0927 13:53:32.722839 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:32.722843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:32.722859 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:53:32.722900 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:32.722903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:32.722963 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:32.723046 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:32.724888 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.725727 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.727624 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.728072 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.728168 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.730850 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:32.730857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:32.730861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:32.730913 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:32.733780 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:32.734705 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:32.734710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:32.734713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:32.734834 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:32.735243 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:32.735247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:32.735250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:32.735503 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:32.735681 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:32.736124 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 13:53:32.736303 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":576,"threads_started":1}
I20260927 13:53:32.737808 5064 heartbeater.cc:347] Connected to a master server at 127.20.133.126:43389
I20260927 13:53:32.738139 5064 heartbeater.cc:464] Registering TS with master...
I20260927 13:53:32.738855 5064 heartbeater.cc:511] Master 127.20.133.126:43389 requested a full tablet report, sending...
I20260927 13:53:32.740259 4616 ts_manager.cc:194] Registered new tserver with Master: 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769)
I20260927 13:53:32.741123 4616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:46757
I20260927 13:53:32.751962 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:53:32.759832 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.752936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.20.133.126:43389)
0927 13:53:32.753111 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:32.753115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:32.753602 (+ 487us) server_negotiation.cc:407] Connection header received
0927 13:53:32.753872 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:32.753877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:32.754107 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:32.754283 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:53:32.754701 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.755773 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.757446 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.758127 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.758254 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.758876 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:53:32.759146 (+ 270us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 13:53:32.759558 (+ 412us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36924: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 13:53:32.759960 5068 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36924: token expired
W20260927 13:53:32.760576 4540 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.133.126:43389)
I20260927 13:53:32.773752 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.761051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.20.133.126:43389)
0927 13:53:32.761250 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:32.761256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:32.761274 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:53:32.761314 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:32.761318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:32.761375 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:32.761452 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:32.762712 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.763419 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.764970 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.765371 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.765475 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.768153 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:32.768158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:32.768161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:32.768197 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:32.770949 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:32.771820 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:32.771825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:32.771827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:32.771977 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:32.772383 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:32.772388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:32.772391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:32.772773 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:32.772977 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:32.773294 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 13:53:32.773500 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:53:32.777292 4616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:53:32.784921 4616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:53:32.801595 4801 heartbeater.cc:503] Master 127.20.133.126:43389 was elected leader, sending a full tablet report...
I20260927 13:53:32.824126 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.797347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.20.133.65:40485)
0927 13:53:32.815154 (+ 17807us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:32.815162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:32.815185 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:53:32.815275 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:32.815281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:32.815458 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:32.815632 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:53:32.817209 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.818992 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.821432 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.822940 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.823078 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.823353 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:32.823711 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 13:53:32.823862 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17628,"thread_start_us":17454,"threads_started":1}
I20260927 13:53:32.824224 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.797159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40485 (local address 127.0.0.1:36942)
0927 13:53:32.812947 (+ 15788us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:32.812997 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:32.813352 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:32.815896 (+ 2544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:32.815904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:32.815984 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:53:32.817015 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.817028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.819126 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.819132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.821237 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.821249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.823162 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.823169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.823387 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.823447 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 13:53:32.823584 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 13:53:32.823792 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15505,"mutex_wait_us":18,"spinlock_wait_cycles":7168,"thread_start_us":15316,"threads_started":1}
I20260927 13:53:32.825493 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.800030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:45449 (local address 127.0.0.1:37026)
0927 13:53:32.814726 (+ 14696us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:32.814748 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:32.814867 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:32.815898 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:32.815905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:32.815948 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:53:32.816708 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.816726 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.820719 (+ 3993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.820724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.822532 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.822546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.824520 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.824527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.824821 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.824851 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:53:32.824947 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 13:53:32.825084 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14566,"mutex_wait_us":79,"spinlock_wait_cycles":3456,"thread_start_us":13731,"threads_started":1}
I20260927 13:53:32.825738 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.797706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.20.133.67:36769)
0927 13:53:32.813169 (+ 15463us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:32.813182 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:32.814996 (+ 1814us) server_negotiation.cc:407] Connection header received
0927 13:53:32.815102 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:32.815110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:32.815344 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:32.815547 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:53:32.817032 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.818514 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.821358 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.822939 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.823091 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.823352 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:32.825221 (+ 1869us) server_negotiation.cc:299] Negotiation successful
0927 13:53:32.825408 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15236,"thread_start_us":15489,"threads_started":1}
I20260927 13:53:32.826303 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.800719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.20.133.66:45449)
0927 13:53:32.805628 (+ 4909us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:32.805640 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:32.814877 (+ 9237us) server_negotiation.cc:407] Connection header received
0927 13:53:32.814998 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:32.815006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:32.815306 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:32.815537 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:53:32.816978 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.818987 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.822719 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.824068 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.824221 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.824443 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:32.825729 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0927 13:53:32.825982 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4667,"thread_start_us":4521,"threads_started":1}
I20260927 13:53:32.826248 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.797447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:36769 (local address 127.0.0.1:55420)
0927 13:53:32.814059 (+ 16612us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:32.814083 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:32.814227 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:32.815897 (+ 1670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:32.815904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:32.816006 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:53:32.816798 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.816811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.818672 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.818682 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.821122 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.821140 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.824556 (+ 3416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.824561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.824795 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.824834 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 13:53:32.824908 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 13:53:32.825044 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16466,"mutex_wait_us":244,"spinlock_wait_cycles":125056,"thread_start_us":17002,"threads_started":1}
I20260927 13:53:32.829587 4736 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5e7784ae654c1bb4c2af75c7d73972 (DEFAULT_TABLE table=test-table [id=8202bfc0b7e94168b081a044bcc23c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:53:32.829627 4999 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5e7784ae654c1bb4c2af75c7d73972 (DEFAULT_TABLE table=test-table [id=8202bfc0b7e94168b081a044bcc23c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:53:32.831321 4736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f5e7784ae654c1bb4c2af75c7d73972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:53:32.831506 4999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f5e7784ae654c1bb4c2af75c7d73972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:53:32.831929 4868 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5e7784ae654c1bb4c2af75c7d73972 (DEFAULT_TABLE table=test-table [id=8202bfc0b7e94168b081a044bcc23c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:53:32.833542 4868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f5e7784ae654c1bb4c2af75c7d73972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:53:32.844432 5080 tablet_bootstrap.cc:492] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: Bootstrap starting.
I20260927 13:53:32.847743 5081 tablet_bootstrap.cc:492] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Bootstrap starting.
I20260927 13:53:32.848670 5080 tablet_bootstrap.cc:654] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: Neither blocks nor log segments found. Creating new log.
I20260927 13:53:32.850065 5080 log.cc:824] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:32.850694 5082 tablet_bootstrap.cc:492] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: Bootstrap starting.
I20260927 13:53:32.851550 5081 tablet_bootstrap.cc:654] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Neither blocks nor log segments found. Creating new log.
I20260927 13:53:32.852960 5081 log.cc:824] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:32.854208 5082 tablet_bootstrap.cc:654] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: Neither blocks nor log segments found. Creating new log.
I20260927 13:53:32.855225 5082 log.cc:824] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:32.855361 5080 tablet_bootstrap.cc:492] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: No bootstrap required, opened a new log
I20260927 13:53:32.855705 5080 ts_tablet_manager.cc:1397] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260927 13:53:32.857482 5081 tablet_bootstrap.cc:492] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: No bootstrap required, opened a new log
I20260927 13:53:32.857745 5082 tablet_bootstrap.cc:492] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: No bootstrap required, opened a new log
I20260927 13:53:32.857791 5081 ts_tablet_manager.cc:1397] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260927 13:53:32.858002 5082 ts_tablet_manager.cc:1397] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260927 13:53:32.866139 5082 raft_consensus.cc:348] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:32.866578 5082 raft_consensus.cc:374] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:53:32.866688 5082 raft_consensus.cc:729] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82715806c3854130b0efeb3308034f69, State: Initialized, Role: FOLLOWER
I20260927 13:53:32.866663 5080 raft_consensus.cc:348] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:32.867117 5080 raft_consensus.cc:374] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:53:32.867225 5080 raft_consensus.cc:729] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8345f10771e84351a2d97610722b52d4, State: Initialized, Role: FOLLOWER
I20260927 13:53:32.867360 5082 consensus_queue.cc:260] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:32.867774 5080 consensus_queue.cc:260] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:32.868701 4933 heartbeater.cc:503] Master 127.20.133.126:43389 was elected leader, sending a full tablet report...
I20260927 13:53:32.870723 5082 ts_tablet_manager.cc:1428] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 13:53:32.870862 5081 raft_consensus.cc:348] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:32.871359 5081 raft_consensus.cc:374] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:53:32.871446 5081 raft_consensus.cc:729] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9130609b44d47cc8fa1d0487105b0ff, State: Initialized, Role: FOLLOWER
I20260927 13:53:32.871692 5064 heartbeater.cc:503] Master 127.20.133.126:43389 was elected leader, sending a full tablet report...
I20260927 13:53:32.872095 5081 consensus_queue.cc:260] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:32.873903 5080 ts_tablet_manager.cc:1428] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260927 13:53:32.876835 5081 ts_tablet_manager.cc:1428] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Time spent starting tablet: real 0.019s user 0.013s sys 0.003s
W20260927 13:53:32.965386 5065 tablet.cc:2369] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:53:32.978163 5088 raft_consensus.cc:482] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:53:32.978507 5088 raft_consensus.cc:504] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:32.980496 5088 leader_election.cc:288] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769), 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449)
W20260927 13:53:32.986483 4934 tablet.cc:2369] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:53:32.988961 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.981180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:48675 (local address 127.20.133.66:45449)
0927 13:53:32.981438 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:32.981445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:32.982212 (+ 767us) server_negotiation.cc:407] Connection header received
0927 13:53:32.982454 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:32.982461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:32.982554 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:32.982654 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:53:32.983364 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.984192 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.986394 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.987730 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.987888 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.987977 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:32.988481 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 13:53:32.988639 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 13:53:32.988950 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.981067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:49115 (local address 127.20.133.67:36769)
0927 13:53:32.981301 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:32.981309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:32.982361 (+ 1052us) server_negotiation.cc:407] Connection header received
0927 13:53:32.982589 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:32.982593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:32.982672 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:32.982775 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:53:32.983519 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.984230 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.986359 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.987697 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.987895 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.987984 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:32.988481 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 13:53:32.988645 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 13:53:32.989109 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.981090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:36769 (local address 127.20.133.65:49115)
0927 13:53:32.982236 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:32.982253 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:32.982421 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:32.982821 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:32.982824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:32.982863 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:53:32.983340 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.983353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.984394 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.984397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.986162 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.986174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.988011 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.988017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.988250 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.988312 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 13:53:32.988451 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 13:53:32.988621 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"mutex_wait_us":83,"thread_start_us":706,"threads_started":1}
I20260927 13:53:32.989387 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:32.980934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:45449 (local address 127.20.133.65:48675)
0927 13:53:32.982061 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:32.982080 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:32.982260 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:32.982692 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:32.982697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:32.982740 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:53:32.983185 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.983214 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.984379 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.984385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.986233 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:32.986251 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:32.987989 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:32.987995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:32.988255 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:32.988309 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 13:53:32.988451 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 13:53:32.988621 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":126,"thread_start_us":640,"threads_started":1}
I20260927 13:53:32.990375 5019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5e7784ae654c1bb4c2af75c7d73972" candidate_uuid: "c9130609b44d47cc8fa1d0487105b0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8345f10771e84351a2d97610722b52d4" is_pre_election: true
I20260927 13:53:32.990474 4888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5e7784ae654c1bb4c2af75c7d73972" candidate_uuid: "c9130609b44d47cc8fa1d0487105b0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82715806c3854130b0efeb3308034f69" is_pre_election: true
I20260927 13:53:32.990926 5019 raft_consensus.cc:2454] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9130609b44d47cc8fa1d0487105b0ff in term 0.
I20260927 13:53:32.991093 4888 raft_consensus.cc:2454] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9130609b44d47cc8fa1d0487105b0ff in term 0.
I20260927 13:53:32.991735 4689 leader_election.cc:302] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8345f10771e84351a2d97610722b52d4, c9130609b44d47cc8fa1d0487105b0ff; no voters:
I20260927 13:53:32.992203 5088 raft_consensus.cc:2781] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:53:32.992416 5088 raft_consensus.cc:482] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:53:32.992530 5088 raft_consensus.cc:3037] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:53:32.994743 5088 raft_consensus.cc:504] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:32.995395 5088 leader_election.cc:288] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 1 election: Requested vote from peers 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769), 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449)
I20260927 13:53:32.996176 5019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5e7784ae654c1bb4c2af75c7d73972" candidate_uuid: "c9130609b44d47cc8fa1d0487105b0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8345f10771e84351a2d97610722b52d4"
I20260927 13:53:32.996428 5019 raft_consensus.cc:3037] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:53:32.996555 4888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5e7784ae654c1bb4c2af75c7d73972" candidate_uuid: "c9130609b44d47cc8fa1d0487105b0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82715806c3854130b0efeb3308034f69"
I20260927 13:53:32.996881 4888 raft_consensus.cc:3037] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:53:32.998870 5019 raft_consensus.cc:2454] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9130609b44d47cc8fa1d0487105b0ff in term 1.
I20260927 13:53:32.999503 4689 leader_election.cc:302] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8345f10771e84351a2d97610722b52d4, c9130609b44d47cc8fa1d0487105b0ff; no voters:
I20260927 13:53:32.999926 5088 raft_consensus.cc:2781] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:53:32.999991 4888 raft_consensus.cc:2454] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9130609b44d47cc8fa1d0487105b0ff in term 1.
I20260927 13:53:33.000671 5088 raft_consensus.cc:686] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 LEADER]: Becoming Leader. State: Replica: c9130609b44d47cc8fa1d0487105b0ff, State: Running, Role: LEADER
I20260927 13:53:33.001371 5088 consensus_queue.cc:237] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:33.007474 4615 catalog_manager.cc:5495] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff reported cstate change: term changed from 0 to 1, leader changed from <none> to c9130609b44d47cc8fa1d0487105b0ff (127.20.133.65). New cstate: current_term: 1 leader_uuid: "c9130609b44d47cc8fa1d0487105b0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } health_report { overall_health: UNKNOWN } } }
W20260927 13:53:33.008715 4802 tablet.cc:2369] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:53:33.068233 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:33.062119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.20.133.65:40485)
0927 13:53:33.062359 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:33.062365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:33.062391 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:53:33.062566 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:33.062570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:33.062679 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:33.062818 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:53:33.063316 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:33.064039 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:33.065715 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:33.066144 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:33.066246 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:33.066921 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:53:33.067536 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:53:33.067675 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:33.067844 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 13:53:33.067941 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 13:53:33.079188 5019 raft_consensus.cc:1261] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4 [term 1 FOLLOWER]: Refusing update from remote peer c9130609b44d47cc8fa1d0487105b0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:53:33.079353 4888 raft_consensus.cc:1261] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69 [term 1 FOLLOWER]: Refusing update from remote peer c9130609b44d47cc8fa1d0487105b0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:53:33.080453 5088 consensus_queue.cc:1035] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:53:33.080926 5091 consensus_queue.cc:1035] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:53:33.112445 4616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:53:33.112833 4616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:53:33.117973 4616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Sending DeleteTablet for 3 replicas of tablet 4f5e7784ae654c1bb4c2af75c7d73972
I20260927 13:53:33.119809 4868 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5e7784ae654c1bb4c2af75c7d73972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:53:33 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:37026
I20260927 13:53:33.119822 4736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5e7784ae654c1bb4c2af75c7d73972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:53:33 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:36942
I20260927 13:53:33.120051 4999 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5e7784ae654c1bb4c2af75c7d73972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:53:33 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:55420
I20260927 13:53:33.121697 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 4653
I20260927 13:53:33.135226 5104 tablet_replica.cc:330] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: stopping tablet replica
I20260927 13:53:33.135959 5104 raft_consensus.cc:2229] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:53:33.136725 5104 raft_consensus.cc:2258] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:53:33.139649 5104 ts_tablet_manager.cc:1905] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:53:33.143203 5105 tablet_replica.cc:330] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: stopping tablet replica
I20260927 13:53:33.143803 5105 raft_consensus.cc:2229] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:53:33.144730 5105 raft_consensus.cc:2258] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:53:33.146613 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:33.123280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.20.133.65:40485)
0927 13:53:33.123503 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:33.123510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:33.123537 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:53:33.123620 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:33.123626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:33.123713 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:33.123872 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:33.125985 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:33.127472 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:33.129839 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:33.130722 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:33.130855 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:33.136675 (+ 5820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:33.136698 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:33.136725 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:33.136797 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:33.141176 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:33.142378 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:33.142388 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:33.142392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:33.142594 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:33.143308 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:33.143313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:33.143316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:33.143861 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:33.144224 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:33.145913 (+ 1689us) server_negotiation.cc:299] Negotiation successful
0927 13:53:33.146217 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 13:53:33.147446 4796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:53:33.149803 5106 tablet_replica.cc:330] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: stopping tablet replica
I20260927 13:53:33.150344 5106 raft_consensus.cc:2229] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:53:33.150920 5106 raft_consensus.cc:2258] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:53:33.152868 5106 ts_tablet_manager.cc:1905] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:53:33.155083 5104 ts_tablet_manager.cc:1918] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:53:33.155485 5104 log.cc:1196] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/4f5e7784ae654c1bb4c2af75c7d73972
I20260927 13:53:33.156663 5104 ts_tablet_manager.cc:1939] T 4f5e7784ae654c1bb4c2af75c7d73972 P 82715806c3854130b0efeb3308034f69: Deleting consensus metadata
I20260927 13:53:33.158761 5106 ts_tablet_manager.cc:1918] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:53:33.158933 5106 log.cc:1196] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/4f5e7784ae654c1bb4c2af75c7d73972
I20260927 13:53:33.159453 5106 ts_tablet_manager.cc:1939] T 4f5e7784ae654c1bb4c2af75c7d73972 P 8345f10771e84351a2d97610722b52d4: Deleting consensus metadata
I20260927 13:53:33.159565 4603 catalog_manager.cc:4841] TS 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449): tablet 4f5e7784ae654c1bb4c2af75c7d73972 (table test-table [id=8202bfc0b7e94168b081a044bcc23c09]) successfully deleted
I20260927 13:53:33.161532 4600 catalog_manager.cc:4841] TS 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769): tablet 4f5e7784ae654c1bb4c2af75c7d73972 (table test-table [id=8202bfc0b7e94168b081a044bcc23c09]) successfully deleted
I20260927 13:53:33.147640 5105 ts_tablet_manager.cc:1905] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:53:33.265134 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 4653
I20260927 13:53:33.281441 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 4805
W20260927 13:53:33.281819 4601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.133.65:40485 (error 108)
I20260927 13:53:33.283979 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:33.283427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40485 (local address 127.0.0.1:36964)
0927 13:53:33.283661 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:33.283804 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:40485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260927 13:53:33.284237 4601 catalog_manager.cc:4584] TS c9130609b44d47cc8fa1d0487105b0ff (127.20.133.65:40485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f5e7784ae654c1bb4c2af75c7d73972: Network error: Client connection negotiation failed: client connection to 127.20.133.65:40485: connect: Connection refused (error 111)
I20260927 13:53:33.301539 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:33.282348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.20.133.66:45449)
0927 13:53:33.282542 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:33.282547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:33.282568 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:53:33.282824 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:33.282831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:33.282930 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:33.283043 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:33.284940 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:33.285706 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:33.287172 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:33.287596 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:33.287710 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:33.295059 (+ 7349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:33.295078 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:33.295095 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:33.295155 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:33.298353 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:33.299246 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:33.299253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:33.299256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:33.299407 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:33.299793 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:33.299797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:33.299799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:33.300263 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:33.300657 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:33.301019 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 13:53:33.301237 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:53:33.302467 4928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:53:33.338053 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:33.337616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40485 (local address 127.0.0.1:36978)
0927 13:53:33.337821 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:33.337910 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:40485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 13:53:33.339556 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:33.339093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40485 (local address 127.0.0.1:36992)
0927 13:53:33.339322 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:33.339411 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:40485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 13:53:33.400755 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:33.400022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40485 (local address 127.0.0.1:37006)
0927 13:53:33.400427 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:33.400558 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:40485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260927 13:53:33.401976 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:33.401566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40485 (local address 127.0.0.1:37016)
0927 13:53:33.401714 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:33.401827 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:40485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 13:53:33.404402 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 4805
I20260927 13:53:33.419077 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 4936
I20260927 13:53:33.435096 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:33.419953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.20.133.67:36769)
0927 13:53:33.420129 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:33.420134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:33.420248 (+ 114us) server_negotiation.cc:407] Connection header received
0927 13:53:33.420494 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:33.420498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:33.420578 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:33.420680 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:33.422259 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:33.423045 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:33.424651 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:33.425086 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:33.425221 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:33.428594 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:33.428613 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:33.428633 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:33.428686 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:33.431411 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:33.432300 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:33.432308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:33.432311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:33.432590 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:33.433018 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:33.433022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:33.433024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:33.433429 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:33.433702 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:33.434326 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 13:53:33.434651 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:53:33.436036 5059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:53:33.483498 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:33.482653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40485 (local address 127.0.0.1:37018)
0927 13:53:33.482969 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:33.483246 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:40485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 13:53:33.485067 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:33.484509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40485 (local address 127.0.0.1:37026)
0927 13:53:33.484775 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:33.484893 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:40485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260927 13:53:33.529132 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 4936
I20260927 13:53:33.543087 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 4583
I20260927 13:53:33.544494 4646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:53:33.663944 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 4583
I20260927 13:53:53.687482 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:43389
--webserver_interface=127.20.133.126
--webserver_port=45851
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:43389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:53:54.081135 5112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:54.081534 5112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:54.081658 5112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:54.090025 5112 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 13:53:54.090173 5112 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 13:53:54.090214 5112 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 13:53:54.090268 5112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:53:54.090363 5112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:54.090467 5112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:54.090524 5112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:53:54.090581 5112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:53:54.102093 5112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:43389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:43389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=45851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:54.103845 5112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:54.105479 5112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:54.119233 5117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:54.119225 5118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:54.119313 5120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:54.120208 5112 server_base.cc:1034] running on GCE node
I20260927 13:53:54.121233 5112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:54.123076 5112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:54.124410 5112 hybrid_clock.cc:648] HybridClock initialized: now 1790517234124285 us; error 46 us; skew 500 ppm
I20260927 13:53:54.124867 5112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:54 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517234, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:54.132778 5112 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:53:54.134859 5112 webserver.cc:460] Webserver started at http://127.20.133.126:45851/ using document root <none> and password file <none>
I20260927 13:53:54.135764 5112 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:54.135892 5112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:54.142786 5112 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:53:54.146687 5127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:54.148468 5112 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:53:54.148708 5112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "54c568ad302f48a5badafa05921a6faa"
format_stamp: "Formatted at 2026-09-27 13:53:08 on dist-test-slave-q25r"
server_key: "7da52ce031dce9f652f0f5d074162f16"
server_key_iv: "83a98f2df62a04f8ce8857e39a32a0e9"
server_key_version: "encryptionkey@0"
I20260927 13:53:54.149853 5112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:54.175537 5112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:54.179723 5112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:54.179992 5112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:54.200058 5112 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:43389
I20260927 13:53:54.200130 5178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:43389 every 8 connection(s)
I20260927 13:53:54.201694 5112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:53:54.207873 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 5112
I20260927 13:53:54.208757 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:40485
--local_ip_for_outbound_sockets=127.20.133.65
--tserver_master_addrs=127.20.133.126:43389
--webserver_port=39895
--webserver_interface=127.20.133.65
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:53:54.210484 5179 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:53:54.214394 5179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Bootstrap starting.
I20260927 13:53:54.244511 5179 log.cc:824] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:54.268815 5179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:53:54.269601 5179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Bootstrap complete.
I20260927 13:53:54.284225 5179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:54.286083 5179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c568ad302f48a5badafa05921a6faa, State: Initialized, Role: FOLLOWER
I20260927 13:53:54.287317 5179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:54.287748 5179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:53:54.287918 5179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:53:54.288164 5179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 2 FOLLOWER]: Advancing to term 3
I20260927 13:53:54.294813 5179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:54.295593 5179 leader_election.cc:302] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54c568ad302f48a5badafa05921a6faa; no voters:
I20260927 13:53:54.296491 5179 leader_election.cc:288] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 13:53:54.296912 5183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 3 FOLLOWER]: Leader election won for term 3
I20260927 13:53:54.298038 5183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [term 3 LEADER]: Becoming Leader. State: Replica: 54c568ad302f48a5badafa05921a6faa, State: Running, Role: LEADER
I20260927 13:53:54.298730 5183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } }
I20260927 13:53:54.298964 5179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:53:54.301707 5185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54c568ad302f48a5badafa05921a6faa. Latest consensus state: current_term: 3 leader_uuid: "54c568ad302f48a5badafa05921a6faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } } }
I20260927 13:53:54.302141 5185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: This master's current role is: LEADER
I20260927 13:53:54.302760 5184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "54c568ad302f48a5badafa05921a6faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c568ad302f48a5badafa05921a6faa" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 43389 } } }
I20260927 13:53:54.303090 5184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa [sys.catalog]: This master's current role is: LEADER
I20260927 13:53:54.314635 5189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:53:54.324755 5189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:53:54.326612 5189 catalog_manager.cc:1259] Loaded cluster ID: 2e4e32fac308450bb6e2bb190d4d91f9
I20260927 13:53:54.326848 5189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:53:54.331987 5189 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:53:54.334159 5189 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Loaded TSK: 0
I20260927 13:53:54.334410 5189 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 54c568ad302f48a5badafa05921a6faa: Loaded TSK: 1
F20260927 13:53:54.335855 5189 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fceb793b9ad google::LogMessage::Fail() at ??:0
@ 0x7fceb793e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fceb793b4de google::LogMessage::Flush() at ??:0
@ 0x7fceb793ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcebb69f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcebb68e73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fcecd36e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcecd36d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcecd3f92a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fcecd3f90c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fcecd4435e9 std::function<>::operator()() at ??:0
@ 0x7fcecd371cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcecd36fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcecd3f7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcecd3f7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fceba587345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fceba58a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fceba58a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fceba565eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcebd83f6db start_thread at ??:0
@ 0x7fceb67d561f clone at ??:0
W20260927 13:53:54.649295 5181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:54.649634 5181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:54.649762 5181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:54.657291 5181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:54.657439 5181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:54.657610 5181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:53:54.667951 5181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:40485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=39895
--tserver_master_addrs=127.20.133.126:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:54.669385 5181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:54.670751 5181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:54.684085 5226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:54.684036 5229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:54.684060 5227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:54.685007 5181 server_base.cc:1034] running on GCE node
I20260927 13:53:54.685806 5181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:54.686823 5181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:54.688053 5181 hybrid_clock.cc:648] HybridClock initialized: now 1790517234687996 us; error 48 us; skew 500 ppm
I20260927 13:53:54.688400 5181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:54 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517234, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:54.691578 5181 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:53:54.693135 5181 webserver.cc:460] Webserver started at http://127.20.133.65:39895/ using document root <none> and password file <none>
I20260927 13:53:54.693832 5181 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:54.693950 5181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:54.699600 5181 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:53:54.702554 5236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:54.703902 5181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:53:54.704056 5181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "c9130609b44d47cc8fa1d0487105b0ff"
format_stamp: "Formatted at 2026-09-27 13:53:08 on dist-test-slave-q25r"
server_key: "a797bc0ecf089d46978b05e4d45cfa9e"
server_key_iv: "c7cd934d6b919066a45a9fead45c3322"
server_key_version: "encryptionkey@0"
I20260927 13:53:54.705058 5181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:54.730170 5181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:54.734751 5181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:54.735015 5181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:54.736127 5181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:53:54.738152 5243 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 13:53:54.744416 5181 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 13:53:54.744599 5181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260927 13:53:54.744740 5181 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 13:53:54.748035 5181 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 13:53:54.748126 5181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260927 13:53:54.748608 5243 tablet_bootstrap.cc:492] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Bootstrap starting.
I20260927 13:53:54.757624 5243 log.cc:824] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Log is configured to *not* fsync() on all Append() calls
I20260927 13:53:54.773728 5243 tablet_bootstrap.cc:492] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:53:54.774567 5243 tablet_bootstrap.cc:492] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Bootstrap complete.
I20260927 13:53:54.775100 5243 ts_tablet_manager.cc:1397] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20260927 13:53:54.784636 5181 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:40485
I20260927 13:53:54.784658 5350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:40485 every 8 connection(s)
I20260927 13:53:54.784894 5243 raft_consensus.cc:348] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:54.786000 5243 raft_consensus.cc:729] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9130609b44d47cc8fa1d0487105b0ff, State: Initialized, Role: FOLLOWER
I20260927 13:53:54.786172 5181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:53:54.786782 5243 consensus_queue.cc:260] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:54.789714 5243 ts_tablet_manager.cc:1428] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260927 13:53:54.791167 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:54.789170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:55817)
0927 13:53:54.790068 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:54.790306 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":382,"threads_started":1}
W20260927 13:53:54.791846 5351 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:43389 (0 consecutive failures): Network error: Failed to ping master at 127.20.133.126:43389: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
I20260927 13:53:54.792879 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:54.792520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:46651)
0927 13:53:54.792686 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:54.792764 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 13:53:54.793933 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:54.793546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:59517)
0927 13:53:54.793724 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:54.793806 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 13:53:54.794170 5351 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:53:54.795446 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 5181
I20260927 13:53:54.796044 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:45449
--local_ip_for_outbound_sockets=127.20.133.66
--tserver_master_addrs=127.20.133.126:43389
--webserver_port=45447
--webserver_interface=127.20.133.66
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:53:55.143026 5355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:55.143358 5355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:55.143492 5355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:55.150990 5355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:55.151118 5355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:55.151270 5355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:53:55.161530 5355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:45449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=45447
--tserver_master_addrs=127.20.133.126:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:55.162885 5355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:55.164240 5355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:55.177691 5360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:55.177757 5361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:55.177729 5363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:55.179122 5355 server_base.cc:1034] running on GCE node
I20260927 13:53:55.179823 5355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:55.180882 5355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:55.182114 5355 hybrid_clock.cc:648] HybridClock initialized: now 1790517235182059 us; error 48 us; skew 500 ppm
I20260927 13:53:55.182472 5355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:55 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517235, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:55.185652 5355 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:53:55.187155 5355 webserver.cc:460] Webserver started at http://127.20.133.66:45447/ using document root <none> and password file <none>
I20260927 13:53:55.187875 5355 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:55.187980 5355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:55.194098 5355 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 13:53:55.197021 5370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:55.198529 5355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:53:55.198727 5355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "82715806c3854130b0efeb3308034f69"
format_stamp: "Formatted at 2026-09-27 13:53:09 on dist-test-slave-q25r"
server_key: "2e6d18b680044c480dcc63fe0516a3c1"
server_key_iv: "62ccf9efcdb68ee45922c27938220b78"
server_key_version: "encryptionkey@0"
I20260927 13:53:55.199783 5355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:55.222520 5355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:55.226877 5355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:55.227140 5355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:55.228159 5355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:53:55.229606 5355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:53:55.229696 5355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:55.229802 5355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:53:55.229864 5355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:55.261664 5355 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:45449
I20260927 13:53:55.261734 5482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:45449 every 8 connection(s)
I20260927 13:53:55.263116 5355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:53:55.268034 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:55.266164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:33753)
0927 13:53:55.266928 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:55.267179 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":295,"threads_started":1}
W20260927 13:53:55.268796 5483 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:43389 (0 consecutive failures): Network error: Failed to ping master at 127.20.133.126:43389: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
I20260927 13:53:55.270036 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:55.269619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:45925)
0927 13:53:55.269791 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:55.269887 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 13:53:55.270937 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:55.270537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:55805)
0927 13:53:55.270700 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:55.270799 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 13:53:55.271045 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 5355
W20260927 13:53:55.271225 5483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:53:55.271673 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:36769
--local_ip_for_outbound_sockets=127.20.133.67
--tserver_master_addrs=127.20.133.126:43389
--webserver_port=45427
--webserver_interface=127.20.133.67
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:53:55.616817 5486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:53:55.617151 5486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:53:55.617285 5486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:53:55.625274 5486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:53:55.625427 5486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:53:55.625613 5486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:53:55.636096 5486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:36769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=45427
--tserver_master_addrs=127.20.133.126:43389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:53:55.637517 5486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:53:55.638865 5486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:53:55.653662 5491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:55.653975 5494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:53:55.654030 5492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:53:55.655572 5486 server_base.cc:1034] running on GCE node
I20260927 13:53:55.656472 5486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:53:55.657796 5486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:53:55.659034 5486 hybrid_clock.cc:648] HybridClock initialized: now 1790517235658979 us; error 55 us; skew 500 ppm
I20260927 13:53:55.659431 5486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:53:55 dist-test-slave-q25r krb5kdc[4033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517235, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:53:55.663249 5486 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:53:55.665092 5486 webserver.cc:460] Webserver started at http://127.20.133.67:45427/ using document root <none> and password file <none>
I20260927 13:53:55.665967 5486 fs_manager.cc:359] Metadata directory not provided
I20260927 13:53:55.666095 5486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:53:55.672824 5486 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:53:55.676384 5501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:55.678125 5486 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:53:55.678335 5486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "8345f10771e84351a2d97610722b52d4"
format_stamp: "Formatted at 2026-09-27 13:53:09 on dist-test-slave-q25r"
server_key: "3c91f84f4a895465b02e1b945a8de840"
server_key_iv: "6fd32a4eddcd71fe5a15ffe0e4c2bb63"
server_key_version: "encryptionkey@0"
I20260927 13:53:55.679507 5486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:53:55.698053 5486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:53:55.703017 5486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:53:55.703435 5486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:53:55.704778 5486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:53:55.706463 5486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:53:55.706568 5486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:55.706681 5486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:53:55.706741 5486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:53:55.743218 5486 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:36769
I20260927 13:53:55.743304 5613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:36769 every 8 connection(s)
I20260927 13:53:55.744966 5486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:53:55.747270 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 5486
I20260927 13:53:55.750303 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:55.748212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:57089)
0927 13:53:55.749101 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:55.749395 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":354,"threads_started":1}
W20260927 13:53:55.751003 5614 heartbeater.cc:644] Failed to heartbeat to 127.20.133.126:43389 (0 consecutive failures): Network error: Failed to ping master at 127.20.133.126:43389: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
I20260927 13:53:55.751839 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:55.751509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:47229)
0927 13:53:55.751664 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:55.751731 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 13:53:55.752627 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:55.752300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:37135)
0927 13:53:55.752438 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:55.752514 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 13:53:55.752853 5614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:53:55.795836 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:55.794835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:59959)
0927 13:53:55.795456 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:55.795562 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":298,"threads_started":1}
I20260927 13:53:56.083528 5619 raft_consensus.cc:482] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:53:56.083914 5619 raft_consensus.cc:504] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:56.085937 5619 leader_election.cc:288] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769), 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449)
Sep 27 13:53:56 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517234, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
Sep 27 13:53:56 dist-test-slave-q25r krb5kdc[4033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517234, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:53:56.112259 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:56.086885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:51081 (local address 127.20.133.66:45449)
0927 13:53:56.089155 (+ 2270us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:56.089170 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:56.089213 (+ 43us) server_negotiation.cc:407] Connection header received
0927 13:53:56.090865 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:56.090877 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:56.091362 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:56.091728 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:56.093137 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:56.094430 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:56.096386 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:56.097379 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:56.097549 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:56.103695 (+ 6146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:56.103706 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:56.103726 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:56.103787 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:56.107657 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:56.108882 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:56.108891 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:56.108898 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:56.109133 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:56.109654 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:56.109658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:56.109661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:56.110293 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:56.110494 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:56.111356 (+ 862us) server_negotiation.cc:299] Negotiation successful
0927 13:53:56.111784 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2084,"thread_start_us":1942,"threads_started":1}
I20260927 13:53:56.112282 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:56.086579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:46551 (local address 127.20.133.67:36769)
0927 13:53:56.088305 (+ 1726us) server_negotiation.cc:206] Beginning negotiation
0927 13:53:56.088320 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:53:56.088393 (+ 73us) server_negotiation.cc:407] Connection header received
0927 13:53:56.090058 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:53:56.090067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:53:56.090485 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:53:56.090787 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:53:56.092828 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:56.094095 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:56.096254 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:53:56.097285 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:53:56.097532 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:56.103276 (+ 5744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:53:56.103291 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:53:56.103317 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:53:56.103383 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:53:56.107384 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:56.108882 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:56.108891 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:56.108897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:56.109151 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:53:56.109628 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:53:56.109633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:53:56.109636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:53:56.110226 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:53:56.110418 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 13:53:56.111381 (+ 963us) server_negotiation.cc:299] Negotiation successful
0927 13:53:56.111845 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1547,"thread_start_us":1375,"threads_started":1}
I20260927 13:53:56.112282 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:56.086268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:36769 (local address 127.20.133.65:46551)
0927 13:53:56.086701 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:56.086740 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:56.087460 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:56.091028 (+ 3568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:56.091039 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:56.091706 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:53:56.092606 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:56.092635 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:56.094305 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:56.094310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:56.096098 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:56.096119 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:56.097515 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:56.097522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:56.097942 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:56.098986 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:53:56.099031 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:53:56.102736 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:53:56.107627 (+ 4891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:56.107639 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:56.107660 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:56.108671 (+ 1011us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:56.109287 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:56.109291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:56.109294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:56.109498 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:56.110398 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:53:56.110407 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:53:56.110791 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 13:53:56.111083 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 13:53:56.111766 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":192,"spinlock_wait_cycles":38784}
I20260927 13:53:56.113168 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:56.086381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:45449 (local address 127.20.133.65:51081)
0927 13:53:56.088676 (+ 2295us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:56.088695 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:53:56.088816 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:53:56.091729 (+ 2913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:53:56.091736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:53:56.092592 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:53:56.092932 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:56.092945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:56.094623 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:56.094628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:56.096113 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:53:56.096130 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:53:56.097579 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:53:56.097584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:53:56.097910 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:53:56.098986 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:53:56.099028 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:53:56.103267 (+ 4239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:53:56.107891 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:56.107897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:56.107901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:56.108675 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:56.109247 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:53:56.109252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:53:56.109257 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:53:56.109487 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:53:56.110430 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:53:56.110436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:53:56.110785 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 13:53:56.111081 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 13:53:56.111942 (+ 861us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2123,"mutex_wait_us":458,"spinlock_wait_cycles":26752,"thread_start_us":1947,"threads_started":1}
W20260927 13:53:56.116382 5240 leader_election.cc:341] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449): Not found: Tablet not found: 4f5e7784ae654c1bb4c2af75c7d73972
W20260927 13:53:56.116730 5237 leader_election.cc:341] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769): Not found: Tablet not found: 4f5e7784ae654c1bb4c2af75c7d73972
I20260927 13:53:56.116874 5237 leader_election.cc:302] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9130609b44d47cc8fa1d0487105b0ff; no voters: 82715806c3854130b0efeb3308034f69, 8345f10771e84351a2d97610722b52d4
I20260927 13:53:56.117388 5619 raft_consensus.cc:2726] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:53:56.273274 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:56.272051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:41893)
0927 13:53:56.272759 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:56.272951 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":343,"threads_started":1}
I20260927 13:53:56.754516 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:56.753477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:58191)
0927 13:53:56.754043 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:56.754222 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":262,"threads_started":1}
I20260927 13:53:56.797968 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:56.797099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:45165)
0927 13:53:56.797686 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:56.797801 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":272,"threads_started":1}
I20260927 13:53:57.275660 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:57.274601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:56507)
0927 13:53:57.275263 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:57.275418 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":338,"threads_started":1}
I20260927 13:53:57.756415 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:57.755602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:39483)
0927 13:53:57.756102 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:57.756192 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":264,"threads_started":1}
I20260927 13:53:57.799851 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:57.799071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:36505)
0927 13:53:57.799579 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:57.799674 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":249,"threads_started":1}
I20260927 13:53:58.152436 5629 raft_consensus.cc:482] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:53:58.152671 5629 raft_consensus.cc:504] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:53:58.153365 5629 leader_election.cc:288] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769), 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449)
W20260927 13:53:58.154541 5237 leader_election.cc:341] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769): Not found: Tablet not found: 4f5e7784ae654c1bb4c2af75c7d73972
W20260927 13:53:58.154712 5240 leader_election.cc:341] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449): Not found: Tablet not found: 4f5e7784ae654c1bb4c2af75c7d73972
I20260927 13:53:58.154850 5240 leader_election.cc:302] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9130609b44d47cc8fa1d0487105b0ff; no voters: 82715806c3854130b0efeb3308034f69, 8345f10771e84351a2d97610722b52d4
I20260927 13:53:58.155148 5629 raft_consensus.cc:2726] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:53:58.277577 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:58.276794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:47749)
0927 13:53:58.277299 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:58.277397 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":260,"threads_started":1}
I20260927 13:53:58.758471 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:58.757465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:34965)
0927 13:53:58.758121 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:58.758263 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":327,"threads_started":1}
I20260927 13:53:58.801837 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:58.800855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:36167)
0927 13:53:58.801474 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:58.801600 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":287,"threads_started":1}
I20260927 13:53:59.279750 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:59.278822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:38943)
0927 13:53:59.279427 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:59.279540 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":293,"threads_started":1}
I20260927 13:53:59.760766 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:59.759683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:45525)
0927 13:53:59.760371 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:59.760510 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":300,"threads_started":1}
I20260927 13:53:59.804280 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:53:59.803346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:39375)
0927 13:53:59.803919 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 13:53:59.804018 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":254,"threads_started":1}
I20260927 13:54:00.282076 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:00.281074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:51171)
0927 13:54:00.281726 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:00.281847 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":362,"threads_started":1}
I20260927 13:54:00.670248 5637 raft_consensus.cc:482] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:54:00.670531 5637 raft_consensus.cc:504] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:54:00.671371 5637 leader_election.cc:288] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769), 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449)
W20260927 13:54:00.672775 5240 leader_election.cc:341] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449): Not found: Tablet not found: 4f5e7784ae654c1bb4c2af75c7d73972
W20260927 13:54:00.673025 5237 leader_election.cc:341] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769): Not found: Tablet not found: 4f5e7784ae654c1bb4c2af75c7d73972
I20260927 13:54:00.673341 5237 leader_election.cc:302] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9130609b44d47cc8fa1d0487105b0ff; no voters: 82715806c3854130b0efeb3308034f69, 8345f10771e84351a2d97610722b52d4
I20260927 13:54:00.673787 5637 raft_consensus.cc:2726] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:54:00.762998 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:00.761960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:37639)
0927 13:54:00.762644 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:00.762777 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":349,"threads_started":1}
I20260927 13:54:00.806579 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:00.805514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:42687)
0927 13:54:00.806158 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:00.806299 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":303,"threads_started":1}
I20260927 13:54:01.284595 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:01.283492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:59309)
0927 13:54:01.284203 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:01.284352 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":377,"threads_started":1}
I20260927 13:54:01.765300 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:01.764383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:41341)
0927 13:54:01.764959 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:01.765087 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":285,"threads_started":1}
I20260927 13:54:01.808666 5642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:01.807742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:44253)
0927 13:54:01.808280 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:01.808438 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":267,"threads_started":1}
I20260927 13:54:02.286721 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:02.285684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:60857)
0927 13:54:02.286340 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:02.286463 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":295,"threads_started":1}
I20260927 13:54:02.767670 5644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:02.766341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:53787)
0927 13:54:02.767201 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:02.767381 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":412,"threads_started":1}
I20260927 13:54:02.810597 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:02.809766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:50699)
0927 13:54:02.810324 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:02.810430 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":278,"threads_started":1}
I20260927 13:54:03.289011 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:03.287978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:60335)
0927 13:54:03.288634 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:03.288749 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":346,"threads_started":1}
I20260927 13:54:03.769644 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:03.768838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:59717)
0927 13:54:03.769332 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:03.769428 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":222,"threads_started":1}
I20260927 13:54:03.812523 5648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:03.811620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:58937)
0927 13:54:03.812146 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:03.812255 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":282,"threads_started":1}
I20260927 13:54:04.198693 5649 raft_consensus.cc:482] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:54:04.199061 5649 raft_consensus.cc:504] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:54:04.199764 5649 leader_election.cc:288] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769), 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449)
W20260927 13:54:04.200795 5240 leader_election.cc:341] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449): Not found: Tablet not found: 4f5e7784ae654c1bb4c2af75c7d73972
W20260927 13:54:04.200984 5237 leader_election.cc:341] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769): Not found: Tablet not found: 4f5e7784ae654c1bb4c2af75c7d73972
I20260927 13:54:04.201124 5237 leader_election.cc:302] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9130609b44d47cc8fa1d0487105b0ff; no voters: 82715806c3854130b0efeb3308034f69, 8345f10771e84351a2d97610722b52d4
I20260927 13:54:04.201462 5649 raft_consensus.cc:2726] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:54:04.290940 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:04.290193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:43439)
0927 13:54:04.290691 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:04.290787 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":244,"threads_started":1}
I20260927 13:54:04.771651 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:04.770733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:53489)
0927 13:54:04.771364 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:04.771478 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":319,"threads_started":1}
I20260927 13:54:04.814821 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:04.813884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:59249)
0927 13:54:04.814495 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:04.814601 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":318,"threads_started":1}
I20260927 13:54:05.293320 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:05.292326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:39497)
0927 13:54:05.293031 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:05.293127 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":363,"threads_started":1}
I20260927 13:54:05.773867 5654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:05.772974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:56291)
0927 13:54:05.773543 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:05.773641 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":331,"threads_started":1}
I20260927 13:54:05.817056 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:05.815959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:55329)
0927 13:54:05.816684 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:05.816832 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":402,"threads_started":1}
I20260927 13:54:06.295532 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:06.294692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:35045)
0927 13:54:06.295264 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:06.295364 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":289,"threads_started":1}
I20260927 13:54:06.776142 5657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:06.775155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:38235)
0927 13:54:06.775812 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:06.775931 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":356,"threads_started":1}
I20260927 13:54:06.818858 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:06.818085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:47425)
0927 13:54:06.818588 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:06.818685 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":267,"threads_started":1}
I20260927 13:54:07.111172 5659 raft_consensus.cc:482] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:54:07.111481 5659 raft_consensus.cc:504] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9130609b44d47cc8fa1d0487105b0ff" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40485 } } peers { permanent_uuid: "8345f10771e84351a2d97610722b52d4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36769 } } peers { permanent_uuid: "82715806c3854130b0efeb3308034f69" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 45449 } }
I20260927 13:54:07.112355 5659 leader_election.cc:288] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769), 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449)
W20260927 13:54:07.113581 5237 leader_election.cc:341] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8345f10771e84351a2d97610722b52d4 (127.20.133.67:36769): Not found: Tablet not found: 4f5e7784ae654c1bb4c2af75c7d73972
W20260927 13:54:07.113834 5240 leader_election.cc:341] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 82715806c3854130b0efeb3308034f69 (127.20.133.66:45449): Not found: Tablet not found: 4f5e7784ae654c1bb4c2af75c7d73972
I20260927 13:54:07.113977 5240 leader_election.cc:302] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9130609b44d47cc8fa1d0487105b0ff; no voters: 82715806c3854130b0efeb3308034f69, 8345f10771e84351a2d97610722b52d4
I20260927 13:54:07.114316 5659 raft_consensus.cc:2726] T 4f5e7784ae654c1bb4c2af75c7d73972 P c9130609b44d47cc8fa1d0487105b0ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:54:07.297775 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:07.296683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:45677)
0927 13:54:07.297413 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:07.297528 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":372,"threads_started":1}
I20260927 13:54:07.778411 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:07.777531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:35961)
0927 13:54:07.778095 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:07.778208 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":283,"threads_started":1}
I20260927 13:54:07.821136 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:07.820164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:60599)
0927 13:54:07.820787 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:07.820916 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":333,"threads_started":1}
I20260927 13:54:08.299818 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:08.298893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:39151)
0927 13:54:08.299495 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:08.299612 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":309,"threads_started":1}
I20260927 13:54:08.780685 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:08.779639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:49063)
0927 13:54:08.780289 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:08.780487 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":316,"threads_started":1}
I20260927 13:54:08.823169 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:08.822288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:33201)
0927 13:54:08.822842 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:08.822955 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":268,"threads_started":1}
I20260927 13:54:09.302079 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:09.301046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:57435)
0927 13:54:09.301702 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:09.301824 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":341,"threads_started":1}
I20260927 13:54:09.782972 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:09.781908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:55395)
0927 13:54:09.782592 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:09.782726 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":311,"threads_started":1}
I20260927 13:54:09.825711 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:09.824788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:53753)
0927 13:54:09.825367 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:09.825493 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":277,"threads_started":1}
I20260927 13:54:10.305089 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:10.303845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.66:49915)
0927 13:54:10.304758 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:10.304891 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":473,"threads_started":1}
I20260927 13:54:10.758872 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 5181
I20260927 13:54:10.782063 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:10.760395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.20.133.65:40485)
0927 13:54:10.761024 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:10.761037 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:10.761112 (+ 75us) server_negotiation.cc:407] Connection header received
0927 13:54:10.761225 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:10.761236 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:10.761586 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:10.761755 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:10.763255 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:10.764426 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:10.765819 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:10.766512 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:10.766633 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:10.775058 (+ 8425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:10.775086 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:10.775109 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:10.775177 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:10.778683 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:10.779607 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:10.779616 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:10.779619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:10.779805 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:10.780176 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:10.780181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:10.780183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:10.780704 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:10.781013 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:10.781365 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 13:54:10.781688 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":289,"threads_started":1}
I20260927 13:54:10.783660 5346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:10.785499 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:10.784503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.67:46859)
0927 13:54:10.785085 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:10.785262 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":281,"threads_started":1}
I20260927 13:54:10.882812 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:10.881615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:43389 (local address 127.20.133.65:44415)
0927 13:54:10.882280 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:10.882444 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.126:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":346,"threads_started":1}
I20260927 13:54:10.888521 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 5181
I20260927 13:54:10.904639 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 5355
I20260927 13:54:10.919768 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:10.905546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.20.133.66:45449)
0927 13:54:10.906134 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:10.906140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:10.906161 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:54:10.906239 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:10.906244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:10.906361 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:10.906475 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:10.907922 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:10.909035 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:10.910326 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:10.910833 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:10.910945 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:10.913827 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:10.913833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:10.913836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:10.913882 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:10.917147 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:10.917956 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:10.917960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:10.917963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:10.918102 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:10.918407 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:10.918410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:10.918412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:10.918713 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:10.918957 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:10.919191 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 13:54:10.919412 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":313,"threads_started":1}
I20260927 13:54:10.920676 5478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:11.016321 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 5355
I20260927 13:54:11.035073 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 5486
I20260927 13:54:11.050590 5676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:11.036037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.20.133.67:36769)
0927 13:54:11.036725 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:11.036731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:11.036748 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:54:11.036825 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:11.036829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:11.036914 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:11.037024 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:11.038648 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:11.039552 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:11.041017 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:11.041491 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:11.041611 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:11.044438 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:11.044443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:11.044447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:11.044485 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:11.047573 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:11.048318 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:11.048324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:11.048327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:11.048596 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:11.048963 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:11.048969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:11.048972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:11.049298 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:11.049592 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:11.049892 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 13:54:11.050146 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":366,"threads_started":1}
I20260927 13:54:11.051649 5609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:11.141925 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 5486
2026-09-27T13:54:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64801 ms)
[----------] 30 tests from SecurityITest (303041 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:11 dist-test-slave-q25r krb5kdc[5680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:11 dist-test-slave-q25r krb5kdc[5680](info): set up 2 sockets
Sep 27 13:54:11 dist-test-slave-q25r krb5kdc[5680](info): commencing operation
krb5kdc: starting...
W20260927 13:54:12.196430 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:12 dist-test-slave-q25r krb5kdc[5680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:54:12.461432 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:45269
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:45269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:12.808100 5693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:12.808434 5693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:12.808542 5693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:12.815950 5693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:12.816087 5693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:12.816135 5693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:12.816195 5693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:12.816239 5693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:12.827234 5693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:45269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:45269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:12.828665 5693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:12.829964 5693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:12.841976 5698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:12.841985 5701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:12.842023 5699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:12.842372 5693 server_base.cc:1034] running on GCE node
I20260927 13:54:12.843465 5693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:12.845044 5693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:12.846308 5693 hybrid_clock.cc:648] HybridClock initialized: now 1790517252846239 us; error 50 us; skew 500 ppm
I20260927 13:54:12.846750 5693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:12 dist-test-slave-q25r krb5kdc[5680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517252, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:12.850189 5693 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:54:12.851915 5693 webserver.cc:460] Webserver started at http://127.20.133.126:45029/ using document root <none> and password file <none>
I20260927 13:54:12.852758 5693 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:12.852887 5693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:12.853247 5693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:12.856783 5693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "31d58cfc20984bcdbc239d9aa6f31138"
format_stamp: "Formatted at 2026-09-27 13:54:12 on dist-test-slave-q25r"
server_key: "470c916f8c8ffca7a2de4fb229a366bc"
server_key_iv: "1d6707fe06d275c03c95d7a049ded8d5"
server_key_version: "encryptionkey@0"
I20260927 13:54:12.857849 5693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "31d58cfc20984bcdbc239d9aa6f31138"
format_stamp: "Formatted at 2026-09-27 13:54:12 on dist-test-slave-q25r"
server_key: "470c916f8c8ffca7a2de4fb229a366bc"
server_key_iv: "1d6707fe06d275c03c95d7a049ded8d5"
server_key_version: "encryptionkey@0"
I20260927 13:54:12.864053 5693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260927 13:54:12.868237 5708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:12.870746 5693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:54:12.870960 5693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "31d58cfc20984bcdbc239d9aa6f31138"
format_stamp: "Formatted at 2026-09-27 13:54:12 on dist-test-slave-q25r"
server_key: "470c916f8c8ffca7a2de4fb229a366bc"
server_key_iv: "1d6707fe06d275c03c95d7a049ded8d5"
server_key_version: "encryptionkey@0"
I20260927 13:54:12.871207 5693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:12.894675 5693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:12.899034 5693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:12.899315 5693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:12.918080 5693 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:45269
I20260927 13:54:12.918126 5759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:45269 every 8 connection(s)
I20260927 13:54:12.919443 5693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:12.923291 5760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:12.926930 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 5693
I20260927 13:54:12.927160 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:12.927901 21013 external_mini_cluster.cc:1428] Setting key 6d26bb45a6a5d68d88f4659803894c96
I20260927 13:54:12.936645 5760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138: Bootstrap starting.
I20260927 13:54:12.940495 5760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:12.941880 5760 log.cc:824] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:54:12 dist-test-slave-q25r krb5kdc[5680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:12.946285 5760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138: No bootstrap required, opened a new log
I20260927 13:54:12.953577 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:12.929905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38426 (local address 127.20.133.126:45269)
0927 13:54:12.930681 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:12.930693 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:12.930733 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:54:12.932402 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:12.932427 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:12.932884 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:12.933221 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:12.935029 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:12.936405 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:12.938279 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:12.939039 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:12.939191 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:12.944393 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:12.944405 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:12.944427 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:12.944490 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:12.948534 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:12.949334 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:12.949348 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:12.949358 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:12.949577 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:12.950017 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:12.950022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:12.950026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:12.950781 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:12.951067 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:12.951565 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 13:54:12.952067 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":363,"threads_started":1}
I20260927 13:54:12.956179 5760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d58cfc20984bcdbc239d9aa6f31138" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45269 } }
I20260927 13:54:12.956642 5760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:12.956748 5760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d58cfc20984bcdbc239d9aa6f31138, State: Initialized, Role: FOLLOWER
I20260927 13:54:12.957325 5760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d58cfc20984bcdbc239d9aa6f31138" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45269 } }
I20260927 13:54:12.957756 5760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:12.957902 5760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:12.958171 5760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:12.961189 5760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d58cfc20984bcdbc239d9aa6f31138" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45269 } }
I20260927 13:54:12.961767 5760 leader_election.cc:302] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31d58cfc20984bcdbc239d9aa6f31138; no voters:
I20260927 13:54:12.962378 5760 leader_election.cc:288] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:12.962540 5765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:12.963223 5765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [term 1 LEADER]: Becoming Leader. State: Replica: 31d58cfc20984bcdbc239d9aa6f31138, State: Running, Role: LEADER
I20260927 13:54:12.963883 5765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d58cfc20984bcdbc239d9aa6f31138" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45269 } }
I20260927 13:54:12.964558 5760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:12.966867 5766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31d58cfc20984bcdbc239d9aa6f31138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d58cfc20984bcdbc239d9aa6f31138" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45269 } } }
I20260927 13:54:12.967049 5767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d58cfc20984bcdbc239d9aa6f31138. Latest consensus state: current_term: 1 leader_uuid: "31d58cfc20984bcdbc239d9aa6f31138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d58cfc20984bcdbc239d9aa6f31138" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 45269 } } }
I20260927 13:54:12.967377 5766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:12.967378 5767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:12.969383 5772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:12.975418 5772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:12.984352 5772 catalog_manager.cc:1347] Generated new cluster ID: dfb89484410d4057a1811a725abca2f0
I20260927 13:54:12.984484 5772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:13.001924 5772 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:13.003856 5772 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:13.011513 5772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138: Generated new TSK 0
I20260927 13:54:13.012478 5772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:54:13.081954 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45269
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:13.423122 5786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:13.423446 5786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:13.423585 5786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:13.431169 5786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:13.431290 5786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:13.431440 5786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:54:13.441365 5786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:13.442596 5786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:13.443843 5786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:13.456743 5791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:13.456833 5792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:13.456759 5794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:13.457922 5786 server_base.cc:1034] running on GCE node
I20260927 13:54:13.458498 5786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:13.459443 5786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:13.460634 5786 hybrid_clock.cc:648] HybridClock initialized: now 1790517253460581 us; error 44 us; skew 500 ppm
I20260927 13:54:13.460937 5786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:13 dist-test-slave-q25r krb5kdc[5680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517253, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:13.463858 5786 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:54:13.465416 5786 webserver.cc:460] Webserver started at http://127.20.133.65:33171/ using document root <none> and password file <none>
I20260927 13:54:13.466112 5786 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:13.466217 5786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:13.466481 5786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:13.469550 5786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "f459305d50154c118df1749aa49af6d2"
format_stamp: "Formatted at 2026-09-27 13:54:13 on dist-test-slave-q25r"
server_key: "d9d712da5bf3650984fb0b272723cadd"
server_key_iv: "ec755c87d5bc67f2230ed928c4a064d7"
server_key_version: "encryptionkey@0"
I20260927 13:54:13.470323 5786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "f459305d50154c118df1749aa49af6d2"
format_stamp: "Formatted at 2026-09-27 13:54:13 on dist-test-slave-q25r"
server_key: "d9d712da5bf3650984fb0b272723cadd"
server_key_iv: "ec755c87d5bc67f2230ed928c4a064d7"
server_key_version: "encryptionkey@0"
I20260927 13:54:13.475926 5786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 13:54:13.479488 5801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:13.480835 5786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:54:13.480984 5786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "f459305d50154c118df1749aa49af6d2"
format_stamp: "Formatted at 2026-09-27 13:54:13 on dist-test-slave-q25r"
server_key: "d9d712da5bf3650984fb0b272723cadd"
server_key_iv: "ec755c87d5bc67f2230ed928c4a064d7"
server_key_version: "encryptionkey@0"
I20260927 13:54:13.481164 5786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:13.493543 5786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:13.497833 5786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:13.498113 5786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:13.499073 5786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:13.500382 5786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:13.500452 5786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:13.500559 5786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:13.500613 5786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:13.531634 5786 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:41505
I20260927 13:54:13.531654 5913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:41505 every 8 connection(s)
I20260927 13:54:13.532963 5786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:54:13.540522 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 5786
I20260927 13:54:13.540721 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:54:13.541350 21013 external_mini_cluster.cc:1428] Setting key f3fd38f071d94f23aed1210d0d09e0f7
Sep 27 13:54:13 dist-test-slave-q25r krb5kdc[5680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517253, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:13.553465 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:13.535457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:45437 (local address 127.20.133.126:45269)
0927 13:54:13.535915 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:13.535921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:13.536836 (+ 915us) server_negotiation.cc:407] Connection header received
0927 13:54:13.538686 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:13.538692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:13.538776 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:13.538872 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:13.540525 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:13.541325 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:13.542945 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:13.543393 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:13.543506 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:13.547164 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:13.547170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:13.547173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:13.547209 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:13.549998 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:13.550863 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:13.550868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:13.550871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:13.551009 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:13.551463 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:13.551468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:13.551470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:13.551757 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:13.551896 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:13.552727 (+ 831us) server_negotiation.cc:299] Negotiation successful
0927 13:54:13.552989 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":249,"threads_started":1}
I20260927 13:54:13.554581 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:13.535973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45269 (local address 127.20.133.65:45437)
0927 13:54:13.536678 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:13.536722 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:13.538394 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:13.539130 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:13.539139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:13.539706 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:13.540299 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:13.540318 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:13.541518 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:13.541522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:13.542791 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:13.542809 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:13.543618 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:13.543625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:13.543855 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:13.544561 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:54:13.544586 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:13.546902 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:13.550200 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:13.550210 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:13.550228 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:13.550713 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:13.551125 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:13.551130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:13.551134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:13.551333 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:13.551891 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:13.551900 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:13.552266 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 13:54:13.552581 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 13:54:13.553161 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":284,"threads_started":1}
I20260927 13:54:13.556175 5914 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45269
I20260927 13:54:13.556569 5914 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:13.557457 5914 heartbeater.cc:511] Master 127.20.133.126:45269 requested a full tablet report, sending...
I20260927 13:54:13.559695 5725 ts_manager.cc:194] Registered new tserver with Master: f459305d50154c118df1749aa49af6d2 (127.20.133.65:41505)
I20260927 13:54:13.561177 5725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:45437
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:54:13.595544 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45269
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:13.922739 5920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:13.923439 5920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:13.923630 5920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:13.935885 5920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:13.936034 5920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:13.936225 5920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:54:13.948133 5920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:13.949462 5920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:13.950730 5920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:13.963593 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:13.963551 5925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:13.963773 5928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:13.964166 5920 server_base.cc:1034] running on GCE node
I20260927 13:54:13.965030 5920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:13.965942 5920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:13.967137 5920 hybrid_clock.cc:648] HybridClock initialized: now 1790517253967084 us; error 39 us; skew 500 ppm
I20260927 13:54:13.967463 5920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:13 dist-test-slave-q25r krb5kdc[5680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517253, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:13.970738 5920 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:54:13.972240 5920 webserver.cc:460] Webserver started at http://127.20.133.66:35979/ using document root <none> and password file <none>
I20260927 13:54:13.972954 5920 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:13.973075 5920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:13.973381 5920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:13.976436 5920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "acea23d789de4635b61f47955be74277"
format_stamp: "Formatted at 2026-09-27 13:54:13 on dist-test-slave-q25r"
server_key: "b19e2ce3d5b7d20f588d1f2ee7e7f7b1"
server_key_iv: "34295e97f49c97f943c62a4da2019246"
server_key_version: "encryptionkey@0"
I20260927 13:54:13.977206 5920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "acea23d789de4635b61f47955be74277"
format_stamp: "Formatted at 2026-09-27 13:54:13 on dist-test-slave-q25r"
server_key: "b19e2ce3d5b7d20f588d1f2ee7e7f7b1"
server_key_iv: "34295e97f49c97f943c62a4da2019246"
server_key_version: "encryptionkey@0"
I20260927 13:54:13.982227 5920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:54:13.985679 5935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:13.986938 5920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:54:13.987089 5920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "acea23d789de4635b61f47955be74277"
format_stamp: "Formatted at 2026-09-27 13:54:13 on dist-test-slave-q25r"
server_key: "b19e2ce3d5b7d20f588d1f2ee7e7f7b1"
server_key_iv: "34295e97f49c97f943c62a4da2019246"
server_key_version: "encryptionkey@0"
I20260927 13:54:13.987278 5920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:14.007594 5920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:14.011453 5920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:14.011708 5920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:14.012704 5920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:14.014078 5920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:14.014158 5920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:14.014273 5920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:14.014328 5920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:14.044960 5920 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:34173
I20260927 13:54:14.045006 6047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:34173 every 8 connection(s)
I20260927 13:54:14.046214 5920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:54:14.049846 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 5920
I20260927 13:54:14.050024 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:54:14.050514 21013 external_mini_cluster.cc:1428] Setting key 9bb406c9ff9df82572a73504cdcddd9b
Sep 27 13:54:14 dist-test-slave-q25r krb5kdc[5680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517253, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:14.067332 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.048801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:48125 (local address 127.20.133.126:45269)
0927 13:54:14.048998 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:14.049002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:14.050236 (+ 1234us) server_negotiation.cc:407] Connection header received
0927 13:54:14.052288 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:14.052300 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:14.052403 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:14.052506 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:14.054091 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.054808 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.056418 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.056854 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.056969 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.060712 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:14.060719 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:14.060724 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:14.060770 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:14.063950 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:14.064857 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:14.064862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:14.064865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:14.065023 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:14.065483 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:14.065488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:14.065490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:14.065845 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:14.065979 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:14.066742 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 13:54:14.066987 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:54:14.068311 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.049268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45269 (local address 127.20.133.66:48125)
0927 13:54:14.050073 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:14.050116 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:14.051956 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:14.052767 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:14.052775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:14.053322 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:14.053921 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.053940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.054957 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.054961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.056241 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.056256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.057095 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.057103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.057335 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.058008 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:54:14.058046 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:14.060433 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:14.064151 (+ 3718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:14.064160 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:14.064175 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:14.064741 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:14.065148 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:14.065152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:14.065154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:14.065357 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:14.065969 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:14.065976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:14.066375 (+ 399us) client_negotiation.cc:769] Sending connection context
0927 13:54:14.066629 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 13:54:14.067071 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":332,"threads_started":1}
I20260927 13:54:14.070013 6048 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45269
I20260927 13:54:14.070372 6048 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:14.071208 6048 heartbeater.cc:511] Master 127.20.133.126:45269 requested a full tablet report, sending...
I20260927 13:54:14.073055 5725 ts_manager.cc:194] Registered new tserver with Master: acea23d789de4635b61f47955be74277 (127.20.133.66:34173)
I20260927 13:54:14.074154 5725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:48125
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:54:14.107329 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45269
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:14.431972 6053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:14.432298 6053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:14.432441 6053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:14.439752 6053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:14.439863 6053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:14.440021 6053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:54:14.449975 6053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:14.451273 6053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:14.452600 6053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:14.466317 6058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:14.466284 6061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:14.466212 6059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:14.467240 6053 server_base.cc:1034] running on GCE node
I20260927 13:54:14.468044 6053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:14.469230 6053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:14.470486 6053 hybrid_clock.cc:648] HybridClock initialized: now 1790517254470426 us; error 34 us; skew 500 ppm
I20260927 13:54:14.470863 6053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:14 dist-test-slave-q25r krb5kdc[5680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517254, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:14.474220 6053 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:54:14.475950 6053 webserver.cc:460] Webserver started at http://127.20.133.67:41799/ using document root <none> and password file <none>
I20260927 13:54:14.476730 6053 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:14.476836 6053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:14.477137 6053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:14.480031 6053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "60c3fcb2e7aa4433a956adf7362fa077"
format_stamp: "Formatted at 2026-09-27 13:54:14 on dist-test-slave-q25r"
server_key: "669c3b618817c0b1932f065450427340"
server_key_iv: "b97514e4823ccf7e7baa0ff4f8fe84a3"
server_key_version: "encryptionkey@0"
I20260927 13:54:14.480825 6053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "60c3fcb2e7aa4433a956adf7362fa077"
format_stamp: "Formatted at 2026-09-27 13:54:14 on dist-test-slave-q25r"
server_key: "669c3b618817c0b1932f065450427340"
server_key_iv: "b97514e4823ccf7e7baa0ff4f8fe84a3"
server_key_version: "encryptionkey@0"
I20260927 13:54:14.486110 6053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:54:14.489771 6068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:14.491112 6053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:54:14.491261 6053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "60c3fcb2e7aa4433a956adf7362fa077"
format_stamp: "Formatted at 2026-09-27 13:54:14 on dist-test-slave-q25r"
server_key: "669c3b618817c0b1932f065450427340"
server_key_iv: "b97514e4823ccf7e7baa0ff4f8fe84a3"
server_key_version: "encryptionkey@0"
I20260927 13:54:14.491443 6053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:14.514290 6053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:14.518517 6053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:14.518796 6053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:14.519790 6053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:14.521397 6053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:14.521490 6053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:14.521612 6053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:14.521675 6053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:14.554216 6053 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:35189
I20260927 13:54:14.554293 6180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:35189 every 8 connection(s)
I20260927 13:54:14.555675 6053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:54:14.562387 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 6053
I20260927 13:54:14.562611 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:54:14.563163 21013 external_mini_cluster.cc:1428] Setting key 4cb6114ba23dea9bb9052c7e7a68596a
I20260927 13:54:14.565052 5914 heartbeater.cc:503] Master 127.20.133.126:45269 was elected leader, sending a full tablet report...
Sep 27 13:54:14 dist-test-slave-q25r krb5kdc[5680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517254, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:14.577026 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.558376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:36347 (local address 127.20.133.126:45269)
0927 13:54:14.558586 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:14.558593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:14.559918 (+ 1325us) server_negotiation.cc:407] Connection header received
0927 13:54:14.562012 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:14.562018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:14.562134 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:14.562231 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:14.564410 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.565497 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.567232 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.567651 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.567757 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.571553 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:14.571560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:14.571565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:14.571609 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:14.574065 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:14.574799 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:14.574802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:14.574804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:14.574929 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:14.575299 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:14.575303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:14.575306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:14.575638 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:14.575764 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:14.576522 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 13:54:14.576737 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:54:14.578270 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.558979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:45269 (local address 127.20.133.67:36347)
0927 13:54:14.559745 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:14.559785 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:14.561683 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:14.562555 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:14.562567 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:14.563390 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:14.564197 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.564225 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.565657 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.565663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.567093 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.567106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.567867 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.567874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.568101 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.568755 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:54:14.568778 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:14.571314 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:14.574212 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:14.574220 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:14.574234 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:14.574699 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:14.575011 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:14.575014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:14.575016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:14.575204 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:14.575770 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:14.575777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:14.576135 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 13:54:14.576451 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 13:54:14.576985 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":285,"threads_started":1}
I20260927 13:54:14.579602 6181 heartbeater.cc:347] Connected to a master server at 127.20.133.126:45269
I20260927 13:54:14.579898 6181 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:14.580626 6181 heartbeater.cc:511] Master 127.20.133.126:45269 requested a full tablet report, sending...
I20260927 13:54:14.582072 5724 ts_manager.cc:194] Registered new tserver with Master: 60c3fcb2e7aa4433a956adf7362fa077 (127.20.133.67:35189)
I20260927 13:54:14.582916 5724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:36347
I20260927 13:54:14.591833 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:54:14.610895 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.597023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.20.133.126:45269)
0927 13:54:14.597231 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:14.597238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:14.597626 (+ 388us) server_negotiation.cc:407] Connection header received
0927 13:54:14.597798 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:14.597803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:14.597888 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:14.598006 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:14.599762 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.600602 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.602173 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.602611 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.602835 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.605489 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:14.605496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:14.605499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:14.605546 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:14.608315 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:14.609099 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:14.609103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:14.609105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:14.609238 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:14.609557 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:14.609561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:14.609563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:14.609908 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:14.610104 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:14.610458 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 13:54:14.610646 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 13:54:14.615548 5724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:54:14.623095 5724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:54:14.661949 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.636392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.20.133.67:35189)
0927 13:54:14.652757 (+ 16365us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:14.652767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:14.652807 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:54:14.652913 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:14.652922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:14.653581 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:14.653820 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:14.655023 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.656510 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.659073 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.660535 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.660651 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.660906 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:14.661436 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 13:54:14.661606 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16176,"thread_start_us":16022,"threads_started":1}
I20260927 13:54:14.662030 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.636198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:35189 (local address 127.0.0.1:55826)
0927 13:54:14.652348 (+ 16150us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:14.652369 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:14.652522 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:14.654190 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:14.654197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:14.654230 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:14.654783 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.654801 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.656673 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.656678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.658904 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.658919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.661060 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.661067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.661237 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.661270 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:54:14.661364 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 13:54:14.661511 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15957,"mutex_wait_us":2,"thread_start_us":15785,"threads_started":1}
I20260927 13:54:14.662360 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.636420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.20.133.66:34173)
0927 13:54:14.653565 (+ 17145us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:14.653572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:14.653600 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:54:14.653689 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:14.653695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:14.653911 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:14.654099 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:14.655053 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.656492 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.658443 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.659824 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.661643 (+ 1819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.661849 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:14.661915 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 13:54:14.662084 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16979,"thread_start_us":18144,"threads_started":1}
I20260927 13:54:14.662766 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.636111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:34173 (local address 127.0.0.1:34236)
0927 13:54:14.648750 (+ 12639us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:14.648808 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:14.649038 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:14.654353 (+ 5315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:14.654357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:14.654381 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:14.654852 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.654861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.656652 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.656657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.658291 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.658302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.661005 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.661009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.661159 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.661192 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:54:14.661287 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 13:54:14.661470 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11969,"mutex_wait_us":6,"thread_start_us":11782,"threads_started":1}
I20260927 13:54:14.663466 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.653170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:41505 (local address 127.0.0.1:43594)
0927 13:54:14.655333 (+ 2163us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:14.655352 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:14.655475 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:14.656024 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:14.656029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:14.656054 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:14.656431 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.656442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.658289 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.658294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.660098 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.660114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.662770 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.662775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.662930 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.662958 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 13:54:14.663042 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 13:54:14.663168 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2040,"thread_start_us":1885,"threads_started":1}
I20260927 13:54:14.663827 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.649565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.20.133.65:41505)
0927 13:54:14.653112 (+ 3547us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:14.653119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:14.655472 (+ 2353us) server_negotiation.cc:407] Connection header received
0927 13:54:14.655623 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:14.655630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:14.655832 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:14.656018 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:14.656587 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.658129 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.660295 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.661693 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.661799 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.662012 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:14.663375 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0927 13:54:14.663534 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3380,"thread_start_us":3205,"threads_started":1}
I20260927 13:54:14.666040 6116 tablet_service.cc:1467] Processing CreateTablet for tablet 80693b95555f49a8a29bfec78fc8ae3d (DEFAULT_TABLE table=test-table [id=a5c08c5404ec4ddda2fd0a02aa47296e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:14.666185 5983 tablet_service.cc:1467] Processing CreateTablet for tablet 80693b95555f49a8a29bfec78fc8ae3d (DEFAULT_TABLE table=test-table [id=a5c08c5404ec4ddda2fd0a02aa47296e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:14.667436 6116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80693b95555f49a8a29bfec78fc8ae3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:14.666996 5849 tablet_service.cc:1467] Processing CreateTablet for tablet 80693b95555f49a8a29bfec78fc8ae3d (DEFAULT_TABLE table=test-table [id=a5c08c5404ec4ddda2fd0a02aa47296e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:14.667500 5983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80693b95555f49a8a29bfec78fc8ae3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:14.668274 5849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80693b95555f49a8a29bfec78fc8ae3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:14.678790 6199 tablet_bootstrap.cc:492] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: Bootstrap starting.
I20260927 13:54:14.681526 6200 tablet_bootstrap.cc:492] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: Bootstrap starting.
I20260927 13:54:14.682332 6199 tablet_bootstrap.cc:654] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:14.683105 6201 tablet_bootstrap.cc:492] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: Bootstrap starting.
I20260927 13:54:14.683892 6199 log.cc:824] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:14.685081 6200 tablet_bootstrap.cc:654] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:14.686314 6200 log.cc:824] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:14.686398 6201 tablet_bootstrap.cc:654] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:14.687500 6201 log.cc:824] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:14.687789 6199 tablet_bootstrap.cc:492] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: No bootstrap required, opened a new log
I20260927 13:54:14.688110 6199 ts_tablet_manager.cc:1397] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 13:54:14.695355 6201 tablet_bootstrap.cc:492] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: No bootstrap required, opened a new log
I20260927 13:54:14.695667 6201 ts_tablet_manager.cc:1397] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260927 13:54:14.700868 6199 raft_consensus.cc:348] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c3fcb2e7aa4433a956adf7362fa077" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35189 } } peers { permanent_uuid: "acea23d789de4635b61f47955be74277" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 34173 } } peers { permanent_uuid: "f459305d50154c118df1749aa49af6d2" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41505 } }
I20260927 13:54:14.701547 6199 raft_consensus.cc:374] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:14.701700 6199 raft_consensus.cc:729] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c3fcb2e7aa4433a956adf7362fa077, State: Initialized, Role: FOLLOWER
I20260927 13:54:14.702412 6199 consensus_queue.cc:260] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c3fcb2e7aa4433a956adf7362fa077" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35189 } } peers { permanent_uuid: "acea23d789de4635b61f47955be74277" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 34173 } } peers { permanent_uuid: "f459305d50154c118df1749aa49af6d2" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41505 } }
I20260927 13:54:14.703183 6200 tablet_bootstrap.cc:492] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: No bootstrap required, opened a new log
I20260927 13:54:14.703519 6200 ts_tablet_manager.cc:1397] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.012s
I20260927 13:54:14.703644 6181 heartbeater.cc:503] Master 127.20.133.126:45269 was elected leader, sending a full tablet report...
I20260927 13:54:14.704282 6199 ts_tablet_manager.cc:1428] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260927 13:54:14.704236 6201 raft_consensus.cc:348] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c3fcb2e7aa4433a956adf7362fa077" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35189 } } peers { permanent_uuid: "acea23d789de4635b61f47955be74277" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 34173 } } peers { permanent_uuid: "f459305d50154c118df1749aa49af6d2" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41505 } }
I20260927 13:54:14.704710 6201 raft_consensus.cc:374] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:14.704805 6201 raft_consensus.cc:729] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f459305d50154c118df1749aa49af6d2, State: Initialized, Role: FOLLOWER
I20260927 13:54:14.705384 6201 consensus_queue.cc:260] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c3fcb2e7aa4433a956adf7362fa077" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35189 } } peers { permanent_uuid: "acea23d789de4635b61f47955be74277" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 34173 } } peers { permanent_uuid: "f459305d50154c118df1749aa49af6d2" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41505 } }
I20260927 13:54:14.709086 6201 ts_tablet_manager.cc:1428] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 13:54:14.712422 6200 raft_consensus.cc:348] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c3fcb2e7aa4433a956adf7362fa077" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35189 } } peers { permanent_uuid: "acea23d789de4635b61f47955be74277" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 34173 } } peers { permanent_uuid: "f459305d50154c118df1749aa49af6d2" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41505 } }
I20260927 13:54:14.712795 6200 raft_consensus.cc:374] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:14.712886 6200 raft_consensus.cc:729] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acea23d789de4635b61f47955be74277, State: Initialized, Role: FOLLOWER
I20260927 13:54:14.713418 6200 consensus_queue.cc:260] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c3fcb2e7aa4433a956adf7362fa077" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35189 } } peers { permanent_uuid: "acea23d789de4635b61f47955be74277" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 34173 } } peers { permanent_uuid: "f459305d50154c118df1749aa49af6d2" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41505 } }
I20260927 13:54:14.714440 6048 heartbeater.cc:503] Master 127.20.133.126:45269 was elected leader, sending a full tablet report...
I20260927 13:54:14.715082 6200 ts_tablet_manager.cc:1428] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 13:54:14.764812 6205 raft_consensus.cc:482] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:54:14.765133 6205 raft_consensus.cc:504] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c3fcb2e7aa4433a956adf7362fa077" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35189 } } peers { permanent_uuid: "acea23d789de4635b61f47955be74277" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 34173 } } peers { permanent_uuid: "f459305d50154c118df1749aa49af6d2" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41505 } }
I20260927 13:54:14.766805 6205 leader_election.cc:288] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acea23d789de4635b61f47955be74277 (127.20.133.66:34173), f459305d50154c118df1749aa49af6d2 (127.20.133.65:41505)
I20260927 13:54:14.774067 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.767089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:34173 (local address 127.20.133.67:51753)
0927 13:54:14.767293 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:14.767319 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:14.767443 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:14.768690 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:14.768696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:14.768733 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:14.768937 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.768947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.770702 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.770708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.772095 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.772109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.773307 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.773313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.773496 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.773522 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 13:54:14.773608 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 13:54:14.773762 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":25}
I20260927 13:54:14.774045 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.767203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:51753 (local address 127.20.133.66:34173)
0927 13:54:14.767392 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:14.767398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:14.767422 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:54:14.767737 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:14.767743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:14.768482 (+ 739us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:14.768628 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:14.769092 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.770348 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.772283 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.773147 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.773329 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.773410 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:14.773646 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 13:54:14.773788 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:54:14.774541 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.767609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:38081 (local address 127.20.133.65:41505)
0927 13:54:14.767850 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:14.767857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:14.769107 (+ 1250us) server_negotiation.cc:407] Connection header received
0927 13:54:14.769266 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:14.769269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:14.769329 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:14.769389 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:14.770006 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.770848 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.772637 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.773602 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.773764 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.773834 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:14.774139 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 13:54:14.774279 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:54:14.774611 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.767564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:41505 (local address 127.20.133.67:38081)
0927 13:54:14.768974 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:14.769000 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:14.769135 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:14.769431 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:14.769435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:14.769461 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:14.769821 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.769832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.771001 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.771010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.772463 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:14.772476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.773823 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.773829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:14.773991 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.774014 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 13:54:14.774099 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 13:54:14.774231 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"mutex_wait_us":1,"thread_start_us":1049,"threads_started":1}
I20260927 13:54:14.775154 6003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80693b95555f49a8a29bfec78fc8ae3d" candidate_uuid: "60c3fcb2e7aa4433a956adf7362fa077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acea23d789de4635b61f47955be74277" is_pre_election: true
I20260927 13:54:14.775818 6003 raft_consensus.cc:2454] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60c3fcb2e7aa4433a956adf7362fa077 in term 0.
I20260927 13:54:14.775745 5869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80693b95555f49a8a29bfec78fc8ae3d" candidate_uuid: "60c3fcb2e7aa4433a956adf7362fa077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f459305d50154c118df1749aa49af6d2" is_pre_election: true
I20260927 13:54:14.776394 5869 raft_consensus.cc:2454] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60c3fcb2e7aa4433a956adf7362fa077 in term 0.
I20260927 13:54:14.776708 6071 leader_election.cc:302] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c3fcb2e7aa4433a956adf7362fa077, acea23d789de4635b61f47955be74277; no voters:
I20260927 13:54:14.777107 6205 raft_consensus.cc:2781] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:54:14.777248 6205 raft_consensus.cc:482] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:54:14.777362 6205 raft_consensus.cc:3037] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:14.779875 6205 raft_consensus.cc:504] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c3fcb2e7aa4433a956adf7362fa077" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35189 } } peers { permanent_uuid: "acea23d789de4635b61f47955be74277" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 34173 } } peers { permanent_uuid: "f459305d50154c118df1749aa49af6d2" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41505 } }
I20260927 13:54:14.780646 6205 leader_election.cc:288] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [CANDIDATE]: Term 1 election: Requested vote from peers acea23d789de4635b61f47955be74277 (127.20.133.66:34173), f459305d50154c118df1749aa49af6d2 (127.20.133.65:41505)
I20260927 13:54:14.781239 6003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80693b95555f49a8a29bfec78fc8ae3d" candidate_uuid: "60c3fcb2e7aa4433a956adf7362fa077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acea23d789de4635b61f47955be74277"
I20260927 13:54:14.781273 5869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80693b95555f49a8a29bfec78fc8ae3d" candidate_uuid: "60c3fcb2e7aa4433a956adf7362fa077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f459305d50154c118df1749aa49af6d2"
I20260927 13:54:14.781454 6003 raft_consensus.cc:3037] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:14.781474 5869 raft_consensus.cc:3037] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:14.784000 6003 raft_consensus.cc:2454] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60c3fcb2e7aa4433a956adf7362fa077 in term 1.
I20260927 13:54:14.784317 5869 raft_consensus.cc:2454] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60c3fcb2e7aa4433a956adf7362fa077 in term 1.
I20260927 13:54:14.784510 6071 leader_election.cc:302] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c3fcb2e7aa4433a956adf7362fa077, acea23d789de4635b61f47955be74277; no voters:
I20260927 13:54:14.784885 6205 raft_consensus.cc:2781] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 1 FOLLOWER]: Leader election won for term 1
W20260927 13:54:14.785281 5915 tablet.cc:2369] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:54:14.785514 6205 raft_consensus.cc:686] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 1 LEADER]: Becoming Leader. State: Replica: 60c3fcb2e7aa4433a956adf7362fa077, State: Running, Role: LEADER
I20260927 13:54:14.786424 6205 consensus_queue.cc:237] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c3fcb2e7aa4433a956adf7362fa077" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35189 } } peers { permanent_uuid: "acea23d789de4635b61f47955be74277" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 34173 } } peers { permanent_uuid: "f459305d50154c118df1749aa49af6d2" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41505 } }
I20260927 13:54:14.791236 5725 catalog_manager.cc:5495] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60c3fcb2e7aa4433a956adf7362fa077 (127.20.133.67). New cstate: current_term: 1 leader_uuid: "60c3fcb2e7aa4433a956adf7362fa077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c3fcb2e7aa4433a956adf7362fa077" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 35189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acea23d789de4635b61f47955be74277" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f459305d50154c118df1749aa49af6d2" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41505 } health_report { overall_health: UNKNOWN } } }
W20260927 13:54:14.798163 6049 tablet.cc:2369] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:54:14.807443 6182 tablet.cc:2369] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:54:14.842132 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.836787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.20.133.67:35189)
0927 13:54:14.837013 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:14.837018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:14.837039 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:54:14.837137 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:14.837141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:14.837233 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:14.837381 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:54:14.837719 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.838475 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.839901 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.840386 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.840509 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.841042 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:54:14.841549 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:54:14.841678 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:14.841784 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 13:54:14.841869 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:54:14.850474 5869 raft_consensus.cc:1261] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2 [term 1 FOLLOWER]: Refusing update from remote peer 60c3fcb2e7aa4433a956adf7362fa077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:14.850497 6003 raft_consensus.cc:1261] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277 [term 1 FOLLOWER]: Refusing update from remote peer 60c3fcb2e7aa4433a956adf7362fa077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:14.851436 6209 consensus_queue.cc:1035] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f459305d50154c118df1749aa49af6d2" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:14.851811 6205 consensus_queue.cc:1035] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acea23d789de4635b61f47955be74277" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:14.864719 6213 mvcc.cc:205] Tried to move back new op lower bound from 7333958675857911808 to 7333958675609489408. Current Snapshot: MvccSnapshot[applied={T|T < 7333958675857911808}]
I20260927 13:54:14.877445 5724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:14.877732 5724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:14.881672 5724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31d58cfc20984bcdbc239d9aa6f31138: Sending DeleteTablet for 3 replicas of tablet 80693b95555f49a8a29bfec78fc8ae3d
I20260927 13:54:14.882925 6116 tablet_service.cc:1514] Processing DeleteTablet for tablet 80693b95555f49a8a29bfec78fc8ae3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:14 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:55826
I20260927 13:54:14.882999 5983 tablet_service.cc:1514] Processing DeleteTablet for tablet 80693b95555f49a8a29bfec78fc8ae3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:14 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:34236
I20260927 13:54:14.883262 5849 tablet_service.cc:1514] Processing DeleteTablet for tablet 80693b95555f49a8a29bfec78fc8ae3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:14 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:43594
I20260927 13:54:14.885313 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 5786
I20260927 13:54:14.893975 6224 tablet_replica.cc:330] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: stopping tablet replica
I20260927 13:54:14.895447 6224 raft_consensus.cc:2229] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:14.896675 6224 raft_consensus.cc:2258] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:14.897507 6222 tablet_replica.cc:330] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: stopping tablet replica
I20260927 13:54:14.898339 6222 raft_consensus.cc:2229] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:54:14.900660 6224 ts_tablet_manager.cc:1905] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:14.901408 6222 raft_consensus.cc:2258] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:14.907421 6223 tablet_replica.cc:330] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: stopping tablet replica
I20260927 13:54:14.908746 6222 ts_tablet_manager.cc:1905] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:14.909286 6223 raft_consensus.cc:2229] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:14.909561 6224 ts_tablet_manager.cc:1918] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:14.909875 6224 log.cc:1196] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/80693b95555f49a8a29bfec78fc8ae3d
I20260927 13:54:14.910532 6224 ts_tablet_manager.cc:1939] T 80693b95555f49a8a29bfec78fc8ae3d P f459305d50154c118df1749aa49af6d2: Deleting consensus metadata
I20260927 13:54:14.910989 6223 raft_consensus.cc:2258] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:14.912569 5711 catalog_manager.cc:4841] TS f459305d50154c118df1749aa49af6d2 (127.20.133.65:41505): tablet 80693b95555f49a8a29bfec78fc8ae3d (table test-table [id=a5c08c5404ec4ddda2fd0a02aa47296e]) successfully deleted
Sep 27 13:54:14 dist-test-slave-q25r krb5kdc[5680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:54:14.914866 6223 ts_tablet_manager.cc:1905] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:14.916599 6222 ts_tablet_manager.cc:1918] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:14.916782 6222 log.cc:1196] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/80693b95555f49a8a29bfec78fc8ae3d
I20260927 13:54:14.917367 6222 ts_tablet_manager.cc:1939] T 80693b95555f49a8a29bfec78fc8ae3d P 60c3fcb2e7aa4433a956adf7362fa077: Deleting consensus metadata
I20260927 13:54:14.922326 6223 ts_tablet_manager.cc:1918] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:14.922588 6223 log.cc:1196] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/80693b95555f49a8a29bfec78fc8ae3d
I20260927 13:54:14.923216 6223 ts_tablet_manager.cc:1939] T 80693b95555f49a8a29bfec78fc8ae3d P acea23d789de4635b61f47955be74277: Deleting consensus metadata
I20260927 13:54:14.923517 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:14.887060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43610 (local address 127.20.133.65:41505)
0927 13:54:14.887269 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:14.887275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:14.900362 (+ 13087us) server_negotiation.cc:407] Connection header received
0927 13:54:14.900945 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:14.900953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:14.901057 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:14.901145 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:14.903135 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.904500 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.907579 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:14.908247 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:14.908394 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:14.916083 (+ 7689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:14.916101 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:14.916118 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:14.916173 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:14.920011 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:14.920980 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:14.920990 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:14.920994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:14.921160 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:14.921576 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:14.921582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:14.921585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:14.922029 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:14.922303 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:14.922877 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 13:54:14.923127 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:54:14.925004 5909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:14.925666 5711 catalog_manager.cc:4841] TS acea23d789de4635b61f47955be74277 (127.20.133.66:34173): tablet 80693b95555f49a8a29bfec78fc8ae3d (table test-table [id=a5c08c5404ec4ddda2fd0a02aa47296e]) successfully deleted
I20260927 13:54:14.931069 5710 catalog_manager.cc:4841] TS 60c3fcb2e7aa4433a956adf7362fa077 (127.20.133.67:35189): tablet 80693b95555f49a8a29bfec78fc8ae3d (table test-table [id=a5c08c5404ec4ddda2fd0a02aa47296e]) successfully deleted
I20260927 13:54:15.076876 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 5786
I20260927 13:54:15.092311 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 5920
Sep 27 13:54:15 dist-test-slave-q25r krb5kdc[5680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:54:15.112684 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:15.093192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.20.133.66:34173)
0927 13:54:15.093412 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:15.093418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:15.093439 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:54:15.093577 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:15.093582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:15.093660 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:15.093742 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:15.094959 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:15.095709 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:15.097200 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:15.097599 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:15.097707 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:15.102348 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:15.102372 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:15.102393 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:15.102456 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:15.108662 (+ 6206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:15.109546 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:15.109557 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:15.109561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:15.109733 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:15.110187 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:15.110193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:15.110196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:15.110743 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:15.111033 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:15.112025 (+ 992us) server_negotiation.cc:299] Negotiation successful
0927 13:54:15.112261 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 13:54:15.113384 6043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:15.217130 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 5920
I20260927 13:54:15.237468 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 6053
Sep 27 13:54:15 dist-test-slave-q25r krb5kdc[5680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:54:15.255393 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:15.238381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.20.133.67:35189)
0927 13:54:15.238546 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:15.238550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:15.238569 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:54:15.238771 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:15.238775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:15.238887 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:15.238984 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:15.240763 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:15.241568 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:15.243426 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:15.243880 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:15.244001 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:15.249415 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:15.249432 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:15.249455 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:15.249509 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:15.252486 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:15.253221 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:15.253229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:15.253231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:15.253410 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:15.253786 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:15.253791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:15.253793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:15.254154 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:15.254476 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:15.254809 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 13:54:15.255034 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:54:15.256157 6176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:15.378892 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 6053
I20260927 13:54:15.394879 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 5693
I20260927 13:54:15.396026 5755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:15.506001 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 5693
2026-09-27T13:54:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4364 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T13:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:15Z Disabled control of system clock
I20260927 13:54:15.581961 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:40361
--webserver_interface=127.20.133.126
--webserver_port=0
--builtin_ntp_servers=127.20.133.84:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:54:15.926070 6233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:15.926378 6233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:15.926460 6233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:15.934243 6233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:15.934366 6233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:15.934420 6233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:15.934500 6233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:15.934562 6233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:15.944031 6233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:40361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:15.945397 6233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:15.947116 6233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:15.959014 6241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:15.959100 6239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:15.959129 6238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:15.959915 6233 server_base.cc:1034] running on GCE node
I20260927 13:54:15.960878 6233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:15.962423 6233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:15.963683 6233 hybrid_clock.cc:648] HybridClock initialized: now 1790517255963632 us; error 47 us; skew 500 ppm
I20260927 13:54:15.964082 6233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:15.970059 6233 webserver.cc:460] Webserver started at http://127.20.133.126:38029/ using document root <none> and password file <none>
I20260927 13:54:15.970813 6233 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:15.970925 6233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:15.971233 6233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:15.974270 6233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "0f221a74cdbc447eb1d19476f679d9f1"
format_stamp: "Formatted at 2026-09-27 13:54:15 on dist-test-slave-q25r"
server_key: "2c9ced79340971fb98dc75f025955f40"
server_key_iv: "81a8168af91ca1b6e539a950d3516357"
server_key_version: "encryptionkey@0"
I20260927 13:54:15.975119 6233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "0f221a74cdbc447eb1d19476f679d9f1"
format_stamp: "Formatted at 2026-09-27 13:54:15 on dist-test-slave-q25r"
server_key: "2c9ced79340971fb98dc75f025955f40"
server_key_iv: "81a8168af91ca1b6e539a950d3516357"
server_key_version: "encryptionkey@0"
I20260927 13:54:15.980605 6233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 13:54:15.984253 6247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:15.985826 6233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:54:15.986018 6233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "0f221a74cdbc447eb1d19476f679d9f1"
format_stamp: "Formatted at 2026-09-27 13:54:15 on dist-test-slave-q25r"
server_key: "2c9ced79340971fb98dc75f025955f40"
server_key_iv: "81a8168af91ca1b6e539a950d3516357"
server_key_version: "encryptionkey@0"
I20260927 13:54:15.986264 6233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:16.024421 6233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:16.025417 6233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:16.025748 6233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:16.045877 6233 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:40361
I20260927 13:54:16.045898 6298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:40361 every 8 connection(s)
I20260927 13:54:16.047631 6233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:16.051628 6299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:16.057415 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 6233
I20260927 13:54:16.057593 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:16.058089 21013 external_mini_cluster.cc:1428] Setting key 06b6c7531e235bd1b2f65fda0fbf756a
I20260927 13:54:16.063900 6299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1: Bootstrap starting.
I20260927 13:54:16.067239 6299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:16.068360 6299 log.cc:824] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:16.071574 6299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1: No bootstrap required, opened a new log
I20260927 13:54:16.071827 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:16.060173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46610 (local address 127.20.133.126:40361)
0927 13:54:16.061285 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:16.061301 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:16.061345 (+ 44us) server_negotiation.cc:407] Connection header received
0927 13:54:16.063070 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:16.063094 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:16.063582 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:16.063935 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:16.064526 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:16.065852 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:16.067227 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:16.067949 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:16.068127 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:16.069337 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:16.069349 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:16.069372 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:16.070017 (+ 645us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:16.070097 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:16.070129 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:16.070215 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:16.070516 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 13:54:16.070854 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":374,"threads_started":1}
I20260927 13:54:16.079968 6299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f221a74cdbc447eb1d19476f679d9f1" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40361 } }
I20260927 13:54:16.080442 6299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:16.080546 6299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f221a74cdbc447eb1d19476f679d9f1, State: Initialized, Role: FOLLOWER
I20260927 13:54:16.081197 6299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f221a74cdbc447eb1d19476f679d9f1" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40361 } }
I20260927 13:54:16.081473 6299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:16.081710 6299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:16.081938 6299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:16.084250 6299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f221a74cdbc447eb1d19476f679d9f1" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40361 } }
I20260927 13:54:16.084913 6299 leader_election.cc:302] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f221a74cdbc447eb1d19476f679d9f1; no voters:
I20260927 13:54:16.085498 6299 leader_election.cc:288] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:16.085625 6304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:16.086395 6304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [term 1 LEADER]: Becoming Leader. State: Replica: 0f221a74cdbc447eb1d19476f679d9f1, State: Running, Role: LEADER
I20260927 13:54:16.086895 6304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f221a74cdbc447eb1d19476f679d9f1" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40361 } }
I20260927 13:54:16.087742 6299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:16.089730 6305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f221a74cdbc447eb1d19476f679d9f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f221a74cdbc447eb1d19476f679d9f1" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40361 } } }
I20260927 13:54:16.089694 6306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f221a74cdbc447eb1d19476f679d9f1. Latest consensus state: current_term: 1 leader_uuid: "0f221a74cdbc447eb1d19476f679d9f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f221a74cdbc447eb1d19476f679d9f1" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40361 } } }
I20260927 13:54:16.090262 6305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:16.090267 6306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:16.091903 6310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:16.097429 6310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:16.106597 6310 catalog_manager.cc:1347] Generated new cluster ID: fa1f8d13b7894598a73b8f8a7407552f
I20260927 13:54:16.106729 6310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:16.123346 6310 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:16.125350 6310 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:16.134508 6310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1: Generated new TSK 0
I20260927 13:54:16.135499 6310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:16.149753 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40361
--builtin_ntp_servers=127.20.133.84:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:54:16.502489 6323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:16.502789 6323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:16.502919 6323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:16.510170 6323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:16.510282 6323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:16.510416 6323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:54:16.519593 6323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:16.520915 6323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:16.522660 6323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:16.535637 6331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:16.535673 6329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:16.536238 6328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:16.536460 6323 server_base.cc:1034] running on GCE node
I20260927 13:54:16.537113 6323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:16.538105 6323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:16.539338 6323 hybrid_clock.cc:648] HybridClock initialized: now 1790517256539280 us; error 48 us; skew 500 ppm
I20260927 13:54:16.539675 6323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:16.542014 6323 webserver.cc:460] Webserver started at http://127.20.133.65:35643/ using document root <none> and password file <none>
I20260927 13:54:16.542722 6323 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:16.542829 6323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:16.543138 6323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:16.546133 6323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "fc55cd88b67444bfb303ed928e8c925b"
format_stamp: "Formatted at 2026-09-27 13:54:16 on dist-test-slave-q25r"
server_key: "703908a10fa83b04d264063ae59d92b7"
server_key_iv: "2159bafd9eb5deabc8f497578439fda9"
server_key_version: "encryptionkey@0"
I20260927 13:54:16.546922 6323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "fc55cd88b67444bfb303ed928e8c925b"
format_stamp: "Formatted at 2026-09-27 13:54:16 on dist-test-slave-q25r"
server_key: "703908a10fa83b04d264063ae59d92b7"
server_key_iv: "2159bafd9eb5deabc8f497578439fda9"
server_key_version: "encryptionkey@0"
I20260927 13:54:16.552012 6323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 13:54:16.555589 6337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:16.557111 6323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 13:54:16.557294 6323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "fc55cd88b67444bfb303ed928e8c925b"
format_stamp: "Formatted at 2026-09-27 13:54:16 on dist-test-slave-q25r"
server_key: "703908a10fa83b04d264063ae59d92b7"
server_key_iv: "2159bafd9eb5deabc8f497578439fda9"
server_key_version: "encryptionkey@0"
I20260927 13:54:16.557548 6323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:16.577404 6323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:16.578389 6323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:16.578773 6323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:16.579909 6323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:16.581741 6323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:16.581849 6323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:16.581960 6323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:16.582028 6323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:16.612742 6323 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:33919
I20260927 13:54:16.612807 6449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:33919 every 8 connection(s)
I20260927 13:54:16.614228 6323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:54:16.618754 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 6323
I20260927 13:54:16.618921 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:54:16.619345 21013 external_mini_cluster.cc:1428] Setting key 5a13228b2582112ef84e2c10cfb7b89d
I20260927 13:54:16.622251 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40361
--builtin_ntp_servers=127.20.133.84:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:54:16.627053 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:16.616935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:43577 (local address 127.20.133.126:40361)
0927 13:54:16.617431 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:16.617448 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:16.618623 (+ 1175us) server_negotiation.cc:407] Connection header received
0927 13:54:16.620617 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:16.620622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:16.620694 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:16.620791 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:16.621995 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:16.622752 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:16.624301 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:16.624731 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:16.624840 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:16.626196 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:16.626201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:16.626204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:16.626226 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:16.626269 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:16.626282 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:16.626356 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:16.626578 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 13:54:16.626684 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":222,"threads_started":1}
I20260927 13:54:16.628065 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:16.617518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:40361 (local address 127.20.133.65:43577)
0927 13:54:16.618422 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:16.618484 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:16.620277 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:16.621040 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:16.621052 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:16.621165 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:16.621820 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:16.621838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:16.622903 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:16.622907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:16.624158 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:16.624171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:16.624951 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:16.624958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:16.625196 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:16.625874 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:54:16.625877 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:16.625919 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:54:16.626012 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:16.626369 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:16.626381 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:16.626389 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:54:16.626553 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 13:54:16.626978 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":286,"threads_started":1}
I20260927 13:54:16.629614 6450 heartbeater.cc:347] Connected to a master server at 127.20.133.126:40361
I20260927 13:54:16.629914 6450 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:16.630659 6450 heartbeater.cc:511] Master 127.20.133.126:40361 requested a full tablet report, sending...
I20260927 13:54:16.632886 6264 ts_manager.cc:194] Registered new tserver with Master: fc55cd88b67444bfb303ed928e8c925b (127.20.133.65:33919)
I20260927 13:54:16.634887 6264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.65:43577
W20260927 13:54:16.949507 6454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:16.949810 6454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:16.949894 6454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:16.957216 6454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:16.957329 6454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:16.957449 6454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:54:16.968267 6454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:16.969754 6454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:16.971593 6454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:16.985137 6460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:16.985204 6459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:16.985175 6462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:16.986747 6454 server_base.cc:1034] running on GCE node
I20260927 13:54:16.987346 6454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:16.988382 6454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:16.989622 6454 hybrid_clock.cc:648] HybridClock initialized: now 1790517256989558 us; error 49 us; skew 500 ppm
I20260927 13:54:16.989943 6454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:16.992506 6454 webserver.cc:460] Webserver started at http://127.20.133.66:42133/ using document root <none> and password file <none>
I20260927 13:54:16.993208 6454 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:16.993301 6454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:16.993603 6454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:16.996495 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "036f3663c72e47eb9284da15ed362100"
format_stamp: "Formatted at 2026-09-27 13:54:16 on dist-test-slave-q25r"
server_key: "f71f2a69961d9fb6ee824b475d63c4d3"
server_key_iv: "444cb411b7cd84b4a5291a7aa2fd60c4"
server_key_version: "encryptionkey@0"
I20260927 13:54:16.997289 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "036f3663c72e47eb9284da15ed362100"
format_stamp: "Formatted at 2026-09-27 13:54:16 on dist-test-slave-q25r"
server_key: "f71f2a69961d9fb6ee824b475d63c4d3"
server_key_iv: "444cb411b7cd84b4a5291a7aa2fd60c4"
server_key_version: "encryptionkey@0"
I20260927 13:54:17.002372 6454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 13:54:17.006546 6468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:17.008240 6454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:54:17.008464 6454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "036f3663c72e47eb9284da15ed362100"
format_stamp: "Formatted at 2026-09-27 13:54:16 on dist-test-slave-q25r"
server_key: "f71f2a69961d9fb6ee824b475d63c4d3"
server_key_iv: "444cb411b7cd84b4a5291a7aa2fd60c4"
server_key_version: "encryptionkey@0"
I20260927 13:54:17.008702 6454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:17.026731 6454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:17.027606 6454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:17.027897 6454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:17.028915 6454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:17.030417 6454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:17.030524 6454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:17.030637 6454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:17.030715 6454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:17.067337 6454 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:40741
I20260927 13:54:17.067394 6580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:40741 every 8 connection(s)
I20260927 13:54:17.068715 6454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:54:17.078676 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 6454
I20260927 13:54:17.078843 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:54:17.079289 21013 external_mini_cluster.cc:1428] Setting key dd350043bc37b59cc4a8616d7749eef9
I20260927 13:54:17.081208 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.071289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:60347 (local address 127.20.133.126:40361)
0927 13:54:17.071514 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:17.071523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:17.072912 (+ 1389us) server_negotiation.cc:407] Connection header received
0927 13:54:17.074666 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:17.074671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:17.074743 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:17.074838 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:17.075882 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.076665 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.078319 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.078751 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.078895 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.080400 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:17.080406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:17.080410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:17.080432 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:17.080469 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:17.080485 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:17.080568 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:17.080798 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 13:54:17.080896 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 13:54:17.082297 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40361
--builtin_ntp_servers=127.20.133.84:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:54:17.082569 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.071785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:40361 (local address 127.20.133.66:60347)
0927 13:54:17.072738 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:17.072774 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:17.074386 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:17.075031 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:17.075039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:17.075134 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:17.075718 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.075735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.076840 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.076845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.078139 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.078153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.079058 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.079065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.079296 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.079985 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:54:17.079988 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:17.080035 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:54:17.080168 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:17.080588 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:17.080596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:17.080602 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 13:54:17.080763 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 13:54:17.081143 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":275,"threads_started":1}
I20260927 13:54:17.084170 6581 heartbeater.cc:347] Connected to a master server at 127.20.133.126:40361
I20260927 13:54:17.084537 6581 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:17.085381 6581 heartbeater.cc:511] Master 127.20.133.126:40361 requested a full tablet report, sending...
I20260927 13:54:17.087031 6264 ts_manager.cc:194] Registered new tserver with Master: 036f3663c72e47eb9284da15ed362100 (127.20.133.66:40741)
I20260927 13:54:17.087872 6264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.66:60347
W20260927 13:54:17.407848 6584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:17.408138 6584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:17.408227 6584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:17.415692 6584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:17.415810 6584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:17.415922 6584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:54:17.425936 6584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:17.427258 6584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:17.429100 6584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:17.441752 6590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:17.441823 6589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:17.441780 6592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:17.442495 6584 server_base.cc:1034] running on GCE node
I20260927 13:54:17.443073 6584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:17.443996 6584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:17.445214 6584 hybrid_clock.cc:648] HybridClock initialized: now 1790517257445155 us; error 46 us; skew 500 ppm
I20260927 13:54:17.445561 6584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:17.447798 6584 webserver.cc:460] Webserver started at http://127.20.133.67:44669/ using document root <none> and password file <none>
I20260927 13:54:17.448529 6584 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:17.448634 6584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:17.448935 6584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:17.451798 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "31b66326ba2f4395be46a54d1d1a17bc"
format_stamp: "Formatted at 2026-09-27 13:54:17 on dist-test-slave-q25r"
server_key: "1db9f4feb9f37b0b2f9ee77b7de0cf22"
server_key_iv: "5f0e15f80d49992bcf328f90ffacbc79"
server_key_version: "encryptionkey@0"
I20260927 13:54:17.452641 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "31b66326ba2f4395be46a54d1d1a17bc"
format_stamp: "Formatted at 2026-09-27 13:54:17 on dist-test-slave-q25r"
server_key: "1db9f4feb9f37b0b2f9ee77b7de0cf22"
server_key_iv: "5f0e15f80d49992bcf328f90ffacbc79"
server_key_version: "encryptionkey@0"
I20260927 13:54:17.458058 6584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:54:17.461633 6598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:17.463016 6584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:54:17.463171 6584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "31b66326ba2f4395be46a54d1d1a17bc"
format_stamp: "Formatted at 2026-09-27 13:54:17 on dist-test-slave-q25r"
server_key: "1db9f4feb9f37b0b2f9ee77b7de0cf22"
server_key_iv: "5f0e15f80d49992bcf328f90ffacbc79"
server_key_version: "encryptionkey@0"
I20260927 13:54:17.463399 6584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:17.503057 6584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:17.503953 6584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:17.504231 6584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:17.505439 6584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:17.506940 6584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:17.507037 6584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:17.507153 6584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:17.507223 6584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:17.538367 6584 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:36313
I20260927 13:54:17.538424 6710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:36313 every 8 connection(s)
I20260927 13:54:17.539803 6584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:54:17.547505 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 6584
I20260927 13:54:17.547677 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:54:17.548161 21013 external_mini_cluster.cc:1428] Setting key 3793ded493d9512105b4cd5157cae508
I20260927 13:54:17.552724 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.542530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:54571 (local address 127.20.133.126:40361)
0927 13:54:17.542713 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:17.542718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:17.544027 (+ 1309us) server_negotiation.cc:407] Connection header received
0927 13:54:17.545766 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:17.545771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:17.545842 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:17.545931 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:17.547030 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.547898 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.549970 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.550572 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.550701 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.551955 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:17.551960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:17.551963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:17.551983 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:17.552014 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:17.552026 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:17.552094 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:17.552386 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 13:54:17.552486 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:54:17.553627 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.542984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:40361 (local address 127.20.133.67:54571)
0927 13:54:17.543873 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:17.543906 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:17.545501 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:17.546162 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:17.546170 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:17.546264 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:17.546821 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.546839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.548056 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.548062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.549821 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.549836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.550759 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.550764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.551028 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.551665 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:54:17.551668 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:17.551710 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:54:17.551786 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:17.552136 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:17.552145 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:17.552152 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 13:54:17.552315 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 13:54:17.552649 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":290,"threads_started":1}
I20260927 13:54:17.554841 6711 heartbeater.cc:347] Connected to a master server at 127.20.133.126:40361
I20260927 13:54:17.555114 6711 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:17.555835 6711 heartbeater.cc:511] Master 127.20.133.126:40361 requested a full tablet report, sending...
I20260927 13:54:17.557194 6264 ts_manager.cc:194] Registered new tserver with Master: 31b66326ba2f4395be46a54d1d1a17bc (127.20.133.67:36313)
I20260927 13:54:17.557950 6264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.67:54571
I20260927 13:54:17.563898 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:54:17.573619 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.567989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.20.133.126:40361)
0927 13:54:17.568209 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:17.568213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:17.568581 (+ 368us) server_negotiation.cc:407] Connection header received
0927 13:54:17.568764 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:17.568768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:17.568847 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:17.568931 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:17.569372 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.570127 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.571443 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.571876 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.571992 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.573091 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:17.573096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:17.573099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:17.573121 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:17.573156 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:17.573168 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:17.573237 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:17.573316 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 13:54:17.573394 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 13:54:17.577558 6264 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:54:17.585047 6264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:54:17.611503 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.597997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:36313 (local address 127.0.0.1:56112)
0927 13:54:17.600552 (+ 2555us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:17.600601 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:17.602124 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:17.603367 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:17.603374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:17.603407 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:17.603885 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.603901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.605925 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.605932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.608437 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.608453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.610374 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.610381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.610685 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.610754 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 13:54:17.610878 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 13:54:17.611068 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2231,"mutex_wait_us":23,"thread_start_us":5707,"threads_started":1}
I20260927 13:54:17.616000 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.598019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:40741 (local address 127.0.0.1:44380)
0927 13:54:17.604029 (+ 6010us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:17.604048 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:17.604280 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:17.606465 (+ 2185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:17.606470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:17.606496 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:17.606900 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.606912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.609938 (+ 3026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.609943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.613058 (+ 3115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.613067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.615171 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.615177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.615350 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.615382 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:54:17.615473 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 13:54:17.615611 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5861,"mutex_wait_us":1124,"thread_start_us":2052,"threads_started":1}
I20260927 13:54:17.616084 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.598578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.20.133.66:40741)
0927 13:54:17.599890 (+ 1312us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:17.599901 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:17.604164 (+ 4263us) server_negotiation.cc:407] Connection header received
0927 13:54:17.605691 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:17.605701 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:17.605992 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:17.606184 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:17.607095 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.608587 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.613238 (+ 4651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.614694 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.614826 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.615050 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:17.615544 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 13:54:17.615747 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":963,"threads_started":1}
I20260927 13:54:17.618631 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.598046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.20.133.67:36313)
0927 13:54:17.602804 (+ 4758us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:17.602810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:17.602840 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:54:17.602915 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:17.602921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:17.603122 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:17.603328 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:17.604145 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.605773 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.608726 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.610157 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.611391 (+ 1234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.611623 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:17.611694 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 13:54:17.618290 (+ 6596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4599,"thread_start_us":4432,"threads_started":1}
I20260927 13:54:17.619814 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.601039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33919 (local address 127.0.0.1:35694)
0927 13:54:17.611456 (+ 10417us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:17.611477 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:17.611610 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:17.612621 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:17.612626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:17.612666 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:17.613037 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.613052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.615565 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.615570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.617203 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.617216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.619078 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.619085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.619268 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.619300 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:54:17.619386 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 13:54:17.619516 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10282,"thread_start_us":10136,"threads_started":1}
I20260927 13:54:17.620055 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.601606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.20.133.65:33919)
0927 13:54:17.611035 (+ 9429us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:17.611044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:17.611597 (+ 553us) server_negotiation.cc:407] Connection header received
0927 13:54:17.611765 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:17.611773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:17.612422 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:17.612599 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:17.614015 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.615394 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.617375 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.618884 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.619014 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.619263 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:17.619436 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 13:54:17.619588 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9260,"thread_start_us":9137,"threads_started":1}
I20260927 13:54:17.621140 6516 tablet_service.cc:1467] Processing CreateTablet for tablet 22a876e0241a43e2a945bc6b9a3f59a4 (DEFAULT_TABLE table=test-table [id=d76ca37ded374d30a73cb47bb3ae0ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:17.622195 6646 tablet_service.cc:1467] Processing CreateTablet for tablet 22a876e0241a43e2a945bc6b9a3f59a4 (DEFAULT_TABLE table=test-table [id=d76ca37ded374d30a73cb47bb3ae0ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:17.622663 6516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a876e0241a43e2a945bc6b9a3f59a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:17.622915 6385 tablet_service.cc:1467] Processing CreateTablet for tablet 22a876e0241a43e2a945bc6b9a3f59a4 (DEFAULT_TABLE table=test-table [id=d76ca37ded374d30a73cb47bb3ae0ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:17.623530 6646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a876e0241a43e2a945bc6b9a3f59a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:17.623937 6385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a876e0241a43e2a945bc6b9a3f59a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:17.635686 6729 tablet_bootstrap.cc:492] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b: Bootstrap starting.
I20260927 13:54:17.638381 6450 heartbeater.cc:503] Master 127.20.133.126:40361 was elected leader, sending a full tablet report...
I20260927 13:54:17.642277 6729 tablet_bootstrap.cc:654] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:17.643736 6729 log.cc:824] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:17.643882 6730 tablet_bootstrap.cc:492] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: Bootstrap starting.
I20260927 13:54:17.645395 6731 tablet_bootstrap.cc:492] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: Bootstrap starting.
I20260927 13:54:17.647392 6729 tablet_bootstrap.cc:492] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b: No bootstrap required, opened a new log
I20260927 13:54:17.647431 6730 tablet_bootstrap.cc:654] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:17.647718 6729 ts_tablet_manager.cc:1397] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260927 13:54:17.648623 6730 log.cc:824] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:17.649698 6731 tablet_bootstrap.cc:654] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:17.650847 6731 log.cc:824] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:17.654301 6731 tablet_bootstrap.cc:492] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: No bootstrap required, opened a new log
I20260927 13:54:17.654304 6730 tablet_bootstrap.cc:492] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: No bootstrap required, opened a new log
I20260927 13:54:17.654624 6731 ts_tablet_manager.cc:1397] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 13:54:17.654624 6730 ts_tablet_manager.cc:1397] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 13:54:17.656486 6729 raft_consensus.cc:348] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b66326ba2f4395be46a54d1d1a17bc" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36313 } } peers { permanent_uuid: "036f3663c72e47eb9284da15ed362100" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40741 } } peers { permanent_uuid: "fc55cd88b67444bfb303ed928e8c925b" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33919 } }
I20260927 13:54:17.656857 6729 raft_consensus.cc:374] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:17.656968 6729 raft_consensus.cc:729] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc55cd88b67444bfb303ed928e8c925b, State: Initialized, Role: FOLLOWER
I20260927 13:54:17.657618 6729 consensus_queue.cc:260] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b66326ba2f4395be46a54d1d1a17bc" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36313 } } peers { permanent_uuid: "036f3663c72e47eb9284da15ed362100" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40741 } } peers { permanent_uuid: "fc55cd88b67444bfb303ed928e8c925b" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33919 } }
I20260927 13:54:17.659389 6729 ts_tablet_manager.cc:1428] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260927 13:54:17.666272 6730 raft_consensus.cc:348] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b66326ba2f4395be46a54d1d1a17bc" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36313 } } peers { permanent_uuid: "036f3663c72e47eb9284da15ed362100" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40741 } } peers { permanent_uuid: "fc55cd88b67444bfb303ed928e8c925b" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33919 } }
I20260927 13:54:17.666272 6731 raft_consensus.cc:348] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b66326ba2f4395be46a54d1d1a17bc" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36313 } } peers { permanent_uuid: "036f3663c72e47eb9284da15ed362100" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40741 } } peers { permanent_uuid: "fc55cd88b67444bfb303ed928e8c925b" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33919 } }
I20260927 13:54:17.666853 6731 raft_consensus.cc:374] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:17.666888 6730 raft_consensus.cc:374] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:17.666973 6731 raft_consensus.cc:729] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b66326ba2f4395be46a54d1d1a17bc, State: Initialized, Role: FOLLOWER
I20260927 13:54:17.666973 6730 raft_consensus.cc:729] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036f3663c72e47eb9284da15ed362100, State: Initialized, Role: FOLLOWER
I20260927 13:54:17.667686 6731 consensus_queue.cc:260] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b66326ba2f4395be46a54d1d1a17bc" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36313 } } peers { permanent_uuid: "036f3663c72e47eb9284da15ed362100" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40741 } } peers { permanent_uuid: "fc55cd88b67444bfb303ed928e8c925b" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33919 } }
I20260927 13:54:17.667677 6730 consensus_queue.cc:260] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b66326ba2f4395be46a54d1d1a17bc" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36313 } } peers { permanent_uuid: "036f3663c72e47eb9284da15ed362100" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40741 } } peers { permanent_uuid: "fc55cd88b67444bfb303ed928e8c925b" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33919 } }
I20260927 13:54:17.669271 6711 heartbeater.cc:503] Master 127.20.133.126:40361 was elected leader, sending a full tablet report...
I20260927 13:54:17.669270 6581 heartbeater.cc:503] Master 127.20.133.126:40361 was elected leader, sending a full tablet report...
I20260927 13:54:17.669822 6730 ts_tablet_manager.cc:1428] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
I20260927 13:54:17.669821 6731 ts_tablet_manager.cc:1428] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: Time spent starting tablet: real 0.015s user 0.006s sys 0.011s
W20260927 13:54:17.791682 6712 tablet.cc:2369] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:54:17.820874 6582 tablet.cc:2369] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:54:17.860616 6736 raft_consensus.cc:482] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:54:17.860904 6736 raft_consensus.cc:504] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b66326ba2f4395be46a54d1d1a17bc" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36313 } } peers { permanent_uuid: "036f3663c72e47eb9284da15ed362100" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40741 } } peers { permanent_uuid: "fc55cd88b67444bfb303ed928e8c925b" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33919 } }
I20260927 13:54:17.862646 6736 leader_election.cc:288] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b66326ba2f4395be46a54d1d1a17bc (127.20.133.67:36313), fc55cd88b67444bfb303ed928e8c925b (127.20.133.65:33919)
W20260927 13:54:17.866686 6451 tablet.cc:2369] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:54:17.869810 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.863099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:43307 (local address 127.20.133.65:33919)
0927 13:54:17.863343 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:17.863350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:17.864125 (+ 775us) server_negotiation.cc:407] Connection header received
0927 13:54:17.864279 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:17.864283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:17.864375 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:17.864462 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:17.865133 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.865882 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.868123 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.868943 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.869104 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.869184 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:17.869475 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 13:54:17.869605 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 13:54:17.869855 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.863094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:59817 (local address 127.20.133.67:36313)
0927 13:54:17.863328 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:17.863335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:17.864062 (+ 727us) server_negotiation.cc:407] Connection header received
0927 13:54:17.864233 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:17.864239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:17.864313 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:17.864416 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:17.865002 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.865795 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.868042 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.868871 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.869014 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.869086 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:17.869410 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 13:54:17.869550 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:54:17.869884 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.862977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:36313 (local address 127.20.133.66:59817)
0927 13:54:17.863941 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:17.863956 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:17.864094 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:17.864450 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:17.864455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:17.864478 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:17.864855 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.864866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.865910 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.865914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.867843 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.867859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.869065 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.869071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.869249 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.869292 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 13:54:17.869361 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 13:54:17.869480 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":63,"thread_start_us":573,"threads_started":1}
I20260927 13:54:17.869935 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.863091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33919 (local address 127.20.133.66:43307)
0927 13:54:17.864005 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:17.864022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:17.864154 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:17.864475 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:17.864479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:17.864510 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:17.864955 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.864969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.866021 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.866025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.867942 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:17.867957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.869131 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.869141 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:17.869317 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.869340 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 13:54:17.869429 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 13:54:17.869561 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":53,"thread_start_us":433,"threads_started":1}
I20260927 13:54:17.870869 6405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a876e0241a43e2a945bc6b9a3f59a4" candidate_uuid: "036f3663c72e47eb9284da15ed362100" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc55cd88b67444bfb303ed928e8c925b" is_pre_election: true
I20260927 13:54:17.870919 6666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a876e0241a43e2a945bc6b9a3f59a4" candidate_uuid: "036f3663c72e47eb9284da15ed362100" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b66326ba2f4395be46a54d1d1a17bc" is_pre_election: true
I20260927 13:54:17.871351 6405 raft_consensus.cc:2454] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 036f3663c72e47eb9284da15ed362100 in term 0.
I20260927 13:54:17.871383 6666 raft_consensus.cc:2454] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 036f3663c72e47eb9284da15ed362100 in term 0.
I20260927 13:54:17.871991 6469 leader_election.cc:302] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 036f3663c72e47eb9284da15ed362100, 31b66326ba2f4395be46a54d1d1a17bc, fc55cd88b67444bfb303ed928e8c925b; no voters:
I20260927 13:54:17.872372 6736 raft_consensus.cc:2781] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:54:17.872505 6736 raft_consensus.cc:482] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:54:17.872601 6736 raft_consensus.cc:3037] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:17.874729 6736 raft_consensus.cc:504] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b66326ba2f4395be46a54d1d1a17bc" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36313 } } peers { permanent_uuid: "036f3663c72e47eb9284da15ed362100" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40741 } } peers { permanent_uuid: "fc55cd88b67444bfb303ed928e8c925b" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33919 } }
I20260927 13:54:17.875324 6736 leader_election.cc:288] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [CANDIDATE]: Term 1 election: Requested vote from peers 31b66326ba2f4395be46a54d1d1a17bc (127.20.133.67:36313), fc55cd88b67444bfb303ed928e8c925b (127.20.133.65:33919)
I20260927 13:54:17.875873 6666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a876e0241a43e2a945bc6b9a3f59a4" candidate_uuid: "036f3663c72e47eb9284da15ed362100" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b66326ba2f4395be46a54d1d1a17bc"
I20260927 13:54:17.875873 6405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a876e0241a43e2a945bc6b9a3f59a4" candidate_uuid: "036f3663c72e47eb9284da15ed362100" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc55cd88b67444bfb303ed928e8c925b"
I20260927 13:54:17.876087 6405 raft_consensus.cc:3037] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:17.876087 6666 raft_consensus.cc:3037] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:17.879212 6405 raft_consensus.cc:2454] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036f3663c72e47eb9284da15ed362100 in term 1.
I20260927 13:54:17.879220 6666 raft_consensus.cc:2454] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 036f3663c72e47eb9284da15ed362100 in term 1.
I20260927 13:54:17.879637 6469 leader_election.cc:302] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036f3663c72e47eb9284da15ed362100, fc55cd88b67444bfb303ed928e8c925b; no voters:
I20260927 13:54:17.879936 6736 raft_consensus.cc:2781] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:17.880430 6736 raft_consensus.cc:686] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 1 LEADER]: Becoming Leader. State: Replica: 036f3663c72e47eb9284da15ed362100, State: Running, Role: LEADER
I20260927 13:54:17.880875 6736 consensus_queue.cc:237] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b66326ba2f4395be46a54d1d1a17bc" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36313 } } peers { permanent_uuid: "036f3663c72e47eb9284da15ed362100" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40741 } } peers { permanent_uuid: "fc55cd88b67444bfb303ed928e8c925b" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33919 } }
I20260927 13:54:17.885689 6263 catalog_manager.cc:5495] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 reported cstate change: term changed from 0 to 1, leader changed from <none> to 036f3663c72e47eb9284da15ed362100 (127.20.133.66). New cstate: current_term: 1 leader_uuid: "036f3663c72e47eb9284da15ed362100" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b66326ba2f4395be46a54d1d1a17bc" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "036f3663c72e47eb9284da15ed362100" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 40741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc55cd88b67444bfb303ed928e8c925b" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33919 } health_report { overall_health: UNKNOWN } } }
I20260927 13:54:17.906448 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.901263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.20.133.66:40741)
0927 13:54:17.901506 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:17.901511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:17.901555 (+ 44us) server_negotiation.cc:407] Connection header received
0927 13:54:17.901653 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:17.901657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:17.901762 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:17.901898 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:54:17.902191 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.902935 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.904413 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.904853 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.904953 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.905355 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:54:17.905913 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:54:17.906028 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:17.906116 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 13:54:17.906186 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 13:54:17.914903 6405 raft_consensus.cc:1261] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b [term 1 FOLLOWER]: Refusing update from remote peer 036f3663c72e47eb9284da15ed362100: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:17.914903 6666 raft_consensus.cc:1261] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc [term 1 FOLLOWER]: Refusing update from remote peer 036f3663c72e47eb9284da15ed362100: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:17.915977 6740 consensus_queue.cc:1035] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc55cd88b67444bfb303ed928e8c925b" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:17.916316 6736 consensus_queue.cc:1035] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b66326ba2f4395be46a54d1d1a17bc" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 36313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:17.929279 6744 mvcc.cc:205] Tried to move back new op lower bound from 7333958688407519232 to 7333958688283930624. Current Snapshot: MvccSnapshot[applied={T|T < 7333958688407519232}]
I20260927 13:54:17.933305 6745 mvcc.cc:205] Tried to move back new op lower bound from 7333958688407519232 to 7333958688283930624. Current Snapshot: MvccSnapshot[applied={T|T < 7333958688407519232}]
I20260927 13:54:17.945389 6263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:17.945693 6263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:17.949776 6263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f221a74cdbc447eb1d19476f679d9f1: Sending DeleteTablet for 3 replicas of tablet 22a876e0241a43e2a945bc6b9a3f59a4
I20260927 13:54:17.950990 6516 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a876e0241a43e2a945bc6b9a3f59a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:17 UTC) from {username='slave'} at 127.0.0.1:44380
I20260927 13:54:17.950987 6646 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a876e0241a43e2a945bc6b9a3f59a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:17 UTC) from {username='slave'} at 127.0.0.1:56112
I20260927 13:54:17.951363 6385 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a876e0241a43e2a945bc6b9a3f59a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:17 UTC) from {username='slave'} at 127.0.0.1:35694
I20260927 13:54:17.951906 6752 tablet_replica.cc:330] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: stopping tablet replica
I20260927 13:54:17.952404 6752 raft_consensus.cc:2229] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:54:17.952702 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 6323
I20260927 13:54:17.953176 6752 raft_consensus.cc:2258] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:17.955261 6752 ts_tablet_manager.cc:1905] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:17.963045 6752 ts_tablet_manager.cc:1918] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:17.963428 6752 log.cc:1196] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/22a876e0241a43e2a945bc6b9a3f59a4
I20260927 13:54:17.964066 6752 ts_tablet_manager.cc:1939] T 22a876e0241a43e2a945bc6b9a3f59a4 P 036f3663c72e47eb9284da15ed362100: Deleting consensus metadata
I20260927 13:54:17.966078 6248 catalog_manager.cc:4841] TS 036f3663c72e47eb9284da15ed362100 (127.20.133.66:40741): tablet 22a876e0241a43e2a945bc6b9a3f59a4 (table test-table [id=d76ca37ded374d30a73cb47bb3ae0ec4]) successfully deleted
I20260927 13:54:17.968850 6754 tablet_replica.cc:330] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b: stopping tablet replica
I20260927 13:54:17.969099 6753 tablet_replica.cc:330] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: stopping tablet replica
I20260927 13:54:17.969430 6754 raft_consensus.cc:2229] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:17.969591 6753 raft_consensus.cc:2229] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:17.970119 6754 raft_consensus.cc:2258] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:17.970146 6753 raft_consensus.cc:2258] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:17.972669 6753 ts_tablet_manager.cc:1905] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:17.972980 6754 ts_tablet_manager.cc:1905] T 22a876e0241a43e2a945bc6b9a3f59a4 P fc55cd88b67444bfb303ed928e8c925b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:17.973812 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:17.954300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.20.133.65:33919)
0927 13:54:17.954510 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:17.954518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:17.961800 (+ 7282us) server_negotiation.cc:407] Connection header received
0927 13:54:17.961999 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:17.962006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:17.962143 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:17.962287 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:17.966833 (+ 4546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.968085 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.969950 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:17.970584 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:17.970709 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:17.972152 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:17.972163 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:17.972180 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:17.972926 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:17.972990 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:17.973016 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:17.973131 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:17.973359 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 13:54:17.973479 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 13:54:17.975159 6445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:17.980454 6753 ts_tablet_manager.cc:1918] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:17.980639 6753 log.cc:1196] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/22a876e0241a43e2a945bc6b9a3f59a4
I20260927 13:54:17.981177 6753 ts_tablet_manager.cc:1939] T 22a876e0241a43e2a945bc6b9a3f59a4 P 31b66326ba2f4395be46a54d1d1a17bc: Deleting consensus metadata
I20260927 13:54:17.983270 6249 catalog_manager.cc:4841] TS 31b66326ba2f4395be46a54d1d1a17bc (127.20.133.67:36313): tablet 22a876e0241a43e2a945bc6b9a3f59a4 (table test-table [id=d76ca37ded374d30a73cb47bb3ae0ec4]) successfully deleted
I20260927 13:54:18.067273 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 6323
W20260927 13:54:18.084790 6248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.133.65:33919 (error 108)
I20260927 13:54:18.084976 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 6454
I20260927 13:54:18.087134 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:18.086620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33919 (local address 127.0.0.1:35726)
0927 13:54:18.086819 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:18.086962 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 13:54:18.087431 6248 catalog_manager.cc:4584] TS fc55cd88b67444bfb303ed928e8c925b (127.20.133.65:33919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22a876e0241a43e2a945bc6b9a3f59a4: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33919: connect: Connection refused (error 111)
I20260927 13:54:18.093600 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:18.085735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44398 (local address 127.20.133.66:40741)
0927 13:54:18.085912 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:18.085918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:18.086072 (+ 154us) server_negotiation.cc:407] Connection header received
0927 13:54:18.086220 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:18.086226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:18.086335 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:18.086427 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:18.087029 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:18.087988 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:18.089690 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:18.090260 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:18.090406 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:18.091728 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:18.091743 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:18.091759 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:18.092541 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:18.092599 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:18.092624 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:18.092733 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:18.092902 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 13:54:18.093037 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:54:18.094452 6576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:18.141726 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:18.140983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33919 (local address 127.0.0.1:35732)
0927 13:54:18.141351 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:18.141501 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 13:54:18.143225 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:18.142756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33919 (local address 127.0.0.1:35742)
0927 13:54:18.142938 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:18.143036 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 13:54:18.204027 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:18.203502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33919 (local address 127.0.0.1:35750)
0927 13:54:18.203735 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:18.203819 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 13:54:18.205839 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:18.205307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33919 (local address 127.0.0.1:35752)
0927 13:54:18.205562 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:18.205670 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 13:54:18.205920 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 6454
I20260927 13:54:18.220312 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 6584
I20260927 13:54:18.227349 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:18.221142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.20.133.67:36313)
0927 13:54:18.221313 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:18.221319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:18.221339 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:54:18.221534 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:18.221538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:18.221616 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:18.221699 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:18.222095 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:18.222806 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:18.224244 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:18.224785 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:18.224918 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:18.226004 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:18.226018 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:18.226033 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:18.226610 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:18.226659 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:18.226679 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:18.226778 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:18.226915 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 13:54:18.227042 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:54:18.228056 6706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:18.288647 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:18.288128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33919 (local address 127.0.0.1:35764)
0927 13:54:18.288358 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:18.288462 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 13:54:18.289776 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:18.289314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33919 (local address 127.0.0.1:35766)
0927 13:54:18.289531 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:18.289627 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 13:54:18.327702 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 6584
I20260927 13:54:18.342701 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 6233
I20260927 13:54:18.343951 6294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:18.455425 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 6233
2026-09-27T13:54:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2946 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:18 dist-test-slave-q25r krb5kdc[6763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:18 dist-test-slave-q25r krb5kdc[6763](info): set up 2 sockets
Sep 27 13:54:18 dist-test-slave-q25r krb5kdc[6763](info): commencing operation
krb5kdc: starting...
W20260927 13:54:19.488299 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:19 dist-test-slave-q25r krb5kdc[6763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:54:19.776989 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:39569
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:39569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:20.102078 6776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:20.102497 6776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:20.102627 6776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:20.110967 6776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:20.111119 6776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:20.111176 6776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:20.111267 6776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:20.111337 6776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:20.122736 6776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:39569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:39569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:20.124606 6776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:20.126220 6776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:20.141337 6782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:20.142328 6781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:20.143571 6784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:20.146628 6776 server_base.cc:1034] running on GCE node
I20260927 13:54:20.147683 6776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:20.149663 6776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:20.150918 6776 hybrid_clock.cc:648] HybridClock initialized: now 1790517260150835 us; error 75 us; skew 500 ppm
I20260927 13:54:20.151363 6776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:20 dist-test-slave-q25r krb5kdc[6763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517260, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:20.154978 6776 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:54:20.156925 6776 webserver.cc:460] Webserver started at http://127.20.133.126:44385/ using document root <none> and password file <none>
I20260927 13:54:20.157812 6776 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:20.157929 6776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:20.158272 6776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:20.161950 6776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "8f3b15feeb284c188229feebbc960ea0"
format_stamp: "Formatted at 2026-09-27 13:54:20 on dist-test-slave-q25r"
server_key: "f80cdb45cfbc68026f174beb647c04dd"
server_key_iv: "25f942e09dc3b3ba457d355394d79ed4"
server_key_version: "encryptionkey@0"
I20260927 13:54:20.162797 6776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "8f3b15feeb284c188229feebbc960ea0"
format_stamp: "Formatted at 2026-09-27 13:54:20 on dist-test-slave-q25r"
server_key: "f80cdb45cfbc68026f174beb647c04dd"
server_key_iv: "25f942e09dc3b3ba457d355394d79ed4"
server_key_version: "encryptionkey@0"
I20260927 13:54:20.169206 6776 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 13:54:20.173615 6791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:20.175448 6776 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:54:20.175632 6776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "8f3b15feeb284c188229feebbc960ea0"
format_stamp: "Formatted at 2026-09-27 13:54:20 on dist-test-slave-q25r"
server_key: "f80cdb45cfbc68026f174beb647c04dd"
server_key_iv: "25f942e09dc3b3ba457d355394d79ed4"
server_key_version: "encryptionkey@0"
I20260927 13:54:20.175866 6776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:20.204224 6776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:20.208940 6776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:20.209196 6776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:20.228258 6776 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:39569
I20260927 13:54:20.228358 6842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:39569 every 8 connection(s)
I20260927 13:54:20.229743 6776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:20.234345 6843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:20.235956 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 6776
I20260927 13:54:20.236132 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:20.236598 21013 external_mini_cluster.cc:1428] Setting key d226f16fe5964228453d61c14e562ef7
I20260927 13:54:20.251506 6843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0: Bootstrap starting.
Sep 27 13:54:20 dist-test-slave-q25r krb5kdc[6763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:20.256101 6843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:20.257617 6843 log.cc:824] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:20.262005 6843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0: No bootstrap required, opened a new log
I20260927 13:54:20.263602 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:20.238741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.20.133.126:39569)
0927 13:54:20.239725 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:20.239743 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:20.239812 (+ 69us) server_negotiation.cc:407] Connection header received
0927 13:54:20.241827 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:20.241862 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:20.242536 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:20.243109 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:20.245339 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:20.247276 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:20.249102 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:20.249866 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:20.250063 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:20.255081 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:20.255091 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:20.255110 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:20.255172 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:20.258640 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:20.259894 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:20.259901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:20.259909 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:20.260092 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:20.260634 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:20.260637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:20.260639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:20.261202 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:20.261438 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:20.261990 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 13:54:20.262416 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":452,"threads_started":1}
I20260927 13:54:20.270951 6843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3b15feeb284c188229feebbc960ea0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39569 } }
I20260927 13:54:20.271438 6843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:20.271546 6843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3b15feeb284c188229feebbc960ea0, State: Initialized, Role: FOLLOWER
I20260927 13:54:20.272217 6843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3b15feeb284c188229feebbc960ea0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39569 } }
I20260927 13:54:20.272523 6843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:20.272668 6843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:20.272841 6843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:20.275308 6843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3b15feeb284c188229feebbc960ea0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39569 } }
I20260927 13:54:20.275913 6843 leader_election.cc:302] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f3b15feeb284c188229feebbc960ea0; no voters:
I20260927 13:54:20.276631 6843 leader_election.cc:288] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:20.276801 6848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:20.277540 6848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 8f3b15feeb284c188229feebbc960ea0, State: Running, Role: LEADER
I20260927 13:54:20.278223 6848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3b15feeb284c188229feebbc960ea0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39569 } }
I20260927 13:54:20.279098 6843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:20.281543 6849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f3b15feeb284c188229feebbc960ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3b15feeb284c188229feebbc960ea0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39569 } } }
I20260927 13:54:20.281647 6850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f3b15feeb284c188229feebbc960ea0. Latest consensus state: current_term: 1 leader_uuid: "8f3b15feeb284c188229feebbc960ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3b15feeb284c188229feebbc960ea0" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 39569 } } }
I20260927 13:54:20.282059 6849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:20.282068 6850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:20.283829 6855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:20.289318 6855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:20.300534 6855 catalog_manager.cc:1347] Generated new cluster ID: 20c48fb829ce4a9395c930e89399a31a
I20260927 13:54:20.300706 6855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:20.315881 6855 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:20.317932 6855 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:20.337390 6855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0: Generated new TSK 0
I20260927 13:54:20.338588 6855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:54:20.415953 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:39569
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:20.806387 6869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:20.806742 6869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:20.806869 6869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:20.815055 6869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:20.815218 6869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:20.815400 6869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:54:20.827284 6869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:39569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:20.829047 6869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:20.830648 6869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:20.845893 6874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:20.845856 6875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:20.846103 6877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:20.846908 6869 server_base.cc:1034] running on GCE node
I20260927 13:54:20.847637 6869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:20.848814 6869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:20.850055 6869 hybrid_clock.cc:648] HybridClock initialized: now 1790517260849967 us; error 72 us; skew 500 ppm
I20260927 13:54:20.850440 6869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:20 dist-test-slave-q25r krb5kdc[6763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517260, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:20.854035 6869 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:54:20.855804 6869 webserver.cc:460] Webserver started at http://127.20.133.65:42831/ using document root <none> and password file <none>
I20260927 13:54:20.856779 6869 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:20.856926 6869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:20.857270 6869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:20.861013 6869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88"
format_stamp: "Formatted at 2026-09-27 13:54:20 on dist-test-slave-q25r"
server_key: "c5273ba953d8310348addadebd5ecdc3"
server_key_iv: "3efc0f2d41474b2add4c549b4e5b7b38"
server_key_version: "encryptionkey@0"
I20260927 13:54:20.861871 6869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88"
format_stamp: "Formatted at 2026-09-27 13:54:20 on dist-test-slave-q25r"
server_key: "c5273ba953d8310348addadebd5ecdc3"
server_key_iv: "3efc0f2d41474b2add4c549b4e5b7b38"
server_key_version: "encryptionkey@0"
I20260927 13:54:20.869724 6869 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 13:54:20.874732 6884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:20.876610 6869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 13:54:20.876809 6869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88"
format_stamp: "Formatted at 2026-09-27 13:54:20 on dist-test-slave-q25r"
server_key: "c5273ba953d8310348addadebd5ecdc3"
server_key_iv: "3efc0f2d41474b2add4c549b4e5b7b38"
server_key_version: "encryptionkey@0"
I20260927 13:54:20.877039 6869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:20.897835 6869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:20.903018 6869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:20.903338 6869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:20.904511 6869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:20.906143 6869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:20.906234 6869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:20.906343 6869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:20.906390 6869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:20.941699 6869 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:39405
I20260927 13:54:20.941802 6996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:39405 every 8 connection(s)
I20260927 13:54:20.943287 6869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:54:20.948882 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 6869
I20260927 13:54:20.949060 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:54:20.949509 21013 external_mini_cluster.cc:1428] Setting key ef0d118379f21b296287f0f49774e7e9
Sep 27 13:54:20 dist-test-slave-q25r krb5kdc[6763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517260, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:20.966915 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:20.946247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:52921 (local address 127.20.133.126:39569)
0927 13:54:20.946768 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:20.946774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:20.947915 (+ 1141us) server_negotiation.cc:407] Connection header received
0927 13:54:20.950251 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:20.950258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:20.950346 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:20.950451 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:20.952511 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:20.953290 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:20.955031 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:20.955488 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:20.955609 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:20.960239 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:20.960245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:20.960249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:20.960294 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:20.963280 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:20.964213 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:20.964218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:20.964221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:20.964403 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:20.964865 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:20.964870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:20.964872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:20.965186 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:20.965355 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:20.966217 (+ 862us) server_negotiation.cc:299] Negotiation successful
0927 13:54:20.966502 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":234,"threads_started":1}
I20260927 13:54:20.968165 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:20.946776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:39569 (local address 127.20.133.65:52921)
0927 13:54:20.947668 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:20.947728 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:20.949908 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:20.950760 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:20.950770 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:20.951524 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:20.952278 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:20.952299 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:20.953473 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:20.953478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:20.954873 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:20.954888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:20.955755 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:20.955762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:20.956037 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:20.956909 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:54:20.956936 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:20.959906 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:20.963500 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:20.963510 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:20.963529 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:20.964072 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:20.964514 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:20.964520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:20.964523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:20.964748 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:20.965318 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:20.965328 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:20.965781 (+ 453us) client_negotiation.cc:769] Sending connection context
0927 13:54:20.966071 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 13:54:20.966755 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":362,"threads_started":1}
I20260927 13:54:20.970204 6997 heartbeater.cc:347] Connected to a master server at 127.20.133.126:39569
I20260927 13:54:20.970599 6997 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:20.971549 6997 heartbeater.cc:511] Master 127.20.133.126:39569 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
I20260927 13:54:20.974546 6808 ts_manager.cc:194] Registered new tserver with Master: 8591a98ab7fa46bf82e8fb6ce1a84e88 (127.20.133.65:39405)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
I20260927 13:54:20.976365 6808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:52921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:54:21.012306 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:39569
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:21.371125 7003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:21.371431 7003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:21.371563 7003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:21.378973 7003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:21.379124 7003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:21.379308 7003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:54:21.389623 7003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:39569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:21.390975 7003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:21.392272 7003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:21.405891 7008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:21.405835 7009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:21.405920 7011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:21.406955 7003 server_base.cc:1034] running on GCE node
I20260927 13:54:21.407596 7003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:21.408648 7003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:21.409857 7003 hybrid_clock.cc:648] HybridClock initialized: now 1790517261409807 us; error 38 us; skew 500 ppm
I20260927 13:54:21.410173 7003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:21 dist-test-slave-q25r krb5kdc[6763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517261, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:21.413338 7003 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:54:21.414903 7003 webserver.cc:460] Webserver started at http://127.20.133.66:44513/ using document root <none> and password file <none>
I20260927 13:54:21.415594 7003 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:21.415711 7003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:21.416005 7003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:21.418987 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "9b87baa4b872462aa67df88f51e4286e"
format_stamp: "Formatted at 2026-09-27 13:54:21 on dist-test-slave-q25r"
server_key: "43fed2cdb7fb8e5d3fb32a890c466734"
server_key_iv: "7b3d3820079214bfe65de9eb4bbd338a"
server_key_version: "encryptionkey@0"
I20260927 13:54:21.419755 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "9b87baa4b872462aa67df88f51e4286e"
format_stamp: "Formatted at 2026-09-27 13:54:21 on dist-test-slave-q25r"
server_key: "43fed2cdb7fb8e5d3fb32a890c466734"
server_key_iv: "7b3d3820079214bfe65de9eb4bbd338a"
server_key_version: "encryptionkey@0"
I20260927 13:54:21.425220 7003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 13:54:21.428855 7018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:21.430202 7003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:54:21.430366 7003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "9b87baa4b872462aa67df88f51e4286e"
format_stamp: "Formatted at 2026-09-27 13:54:21 on dist-test-slave-q25r"
server_key: "43fed2cdb7fb8e5d3fb32a890c466734"
server_key_iv: "7b3d3820079214bfe65de9eb4bbd338a"
server_key_version: "encryptionkey@0"
I20260927 13:54:21.430564 7003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:21.447994 7003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:21.452284 7003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:21.452607 7003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:21.453619 7003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:21.455014 7003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:21.455094 7003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:21.455228 7003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:21.455295 7003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:21.487505 7003 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:44677
I20260927 13:54:21.487572 7130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:44677 every 8 connection(s)
I20260927 13:54:21.488998 7003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:54:21.491320 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 7003
I20260927 13:54:21.491508 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:54:21.492095 21013 external_mini_cluster.cc:1428] Setting key 69d4f8e79dd1a477159900a3266c4d1e
Sep 27 13:54:21 dist-test-slave-q25r krb5kdc[6763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517261, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:21.510310 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:21.491538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:41723 (local address 127.20.133.126:39569)
0927 13:54:21.492427 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:21.492433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:21.495209 (+ 2776us) server_negotiation.cc:407] Connection header received
0927 13:54:21.495269 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:21.495273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:21.495342 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:21.495449 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:21.497171 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:21.497930 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:21.499582 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:21.500024 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:21.500134 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:21.504218 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:21.504225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:21.504229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:21.504271 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:21.507258 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:21.507991 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:21.507996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:21.508000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:21.508157 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:21.508577 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:21.508581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:21.508584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:21.508923 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:21.509086 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:21.509686 (+ 600us) server_negotiation.cc:299] Negotiation successful
0927 13:54:21.509941 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":604,"threads_started":1}
I20260927 13:54:21.511190 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:21.492109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:39569 (local address 127.20.133.66:41723)
0927 13:54:21.492984 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:21.493035 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:21.494901 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:21.495747 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:21.495757 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:21.496324 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:21.496982 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:21.497001 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:21.498118 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:21.498122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:21.499426 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:21.499441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:21.500266 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:21.500274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:21.500567 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:21.501307 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:54:21.501337 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:21.503939 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:21.507406 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:21.507414 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:21.507429 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:21.507866 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:21.508250 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:21.508253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:21.508255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:21.508478 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:21.509026 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:21.509033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:21.509353 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 13:54:21.509573 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 13:54:21.509988 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":364,"threads_started":1}
I20260927 13:54:21.512624 7131 heartbeater.cc:347] Connected to a master server at 127.20.133.126:39569
I20260927 13:54:21.512957 7131 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:21.513865 7131 heartbeater.cc:511] Master 127.20.133.126:39569 requested a full tablet report, sending...
I20260927 13:54:21.515559 6808 ts_manager.cc:194] Registered new tserver with Master: 9b87baa4b872462aa67df88f51e4286e (127.20.133.66:44677)
I20260927 13:54:21.516597 6808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:41723
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:54:21.548749 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:39569
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:21.874639 7137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:21.874931 7137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:21.875052 7137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:21.882625 7137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:21.882825 7137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:21.883020 7137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:54:21.893124 7137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:39569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:21.894469 7137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:21.895725 7137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:21.908877 7142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:21.908828 7143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:21.908895 7145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:21.909694 7137 server_base.cc:1034] running on GCE node
I20260927 13:54:21.910259 7137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:21.911161 7137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:21.912417 7137 hybrid_clock.cc:648] HybridClock initialized: now 1790517261912361 us; error 43 us; skew 500 ppm
I20260927 13:54:21.912765 7137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:21 dist-test-slave-q25r krb5kdc[6763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517261, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:21.915778 7137 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:54:21.917279 7137 webserver.cc:460] Webserver started at http://127.20.133.67:45079/ using document root <none> and password file <none>
I20260927 13:54:21.917953 7137 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:21.918059 7137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:21.918357 7137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:21.921300 7137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "a7c37e6012494755bd560053e2702cc4"
format_stamp: "Formatted at 2026-09-27 13:54:21 on dist-test-slave-q25r"
server_key: "ed5b5f9417859aca2e56a32a8c338518"
server_key_iv: "84394066ab518a0fb64f0a7b80d139a8"
server_key_version: "encryptionkey@0"
I20260927 13:54:21.922082 7137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "a7c37e6012494755bd560053e2702cc4"
format_stamp: "Formatted at 2026-09-27 13:54:21 on dist-test-slave-q25r"
server_key: "ed5b5f9417859aca2e56a32a8c338518"
server_key_iv: "84394066ab518a0fb64f0a7b80d139a8"
server_key_version: "encryptionkey@0"
I20260927 13:54:21.927341 7137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:54:21.931062 7152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:21.932444 7137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:54:21.932596 7137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "a7c37e6012494755bd560053e2702cc4"
format_stamp: "Formatted at 2026-09-27 13:54:21 on dist-test-slave-q25r"
server_key: "ed5b5f9417859aca2e56a32a8c338518"
server_key_iv: "84394066ab518a0fb64f0a7b80d139a8"
server_key_version: "encryptionkey@0"
I20260927 13:54:21.932781 7137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:21.947991 7137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:21.952879 7137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:21.953193 7137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:21.954226 7137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:21.955652 7137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:21.955726 7137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:21.955829 7137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:21.955883 7137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:21.980363 6997 heartbeater.cc:503] Master 127.20.133.126:39569 was elected leader, sending a full tablet report...
I20260927 13:54:21.989920 7137 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:44275
I20260927 13:54:21.989953 7264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:44275 every 8 connection(s)
I20260927 13:54:21.991201 7137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:54:21.992769 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 7137
I20260927 13:54:21.992930 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:54:21.993472 21013 external_mini_cluster.cc:1428] Setting key c77175be3dafb0e0047c8900a619af32
Sep 27 13:54:22 dist-test-slave-q25r krb5kdc[6763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517261, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:22.011478 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:21.993840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:52825 (local address 127.20.133.126:39569)
0927 13:54:21.994033 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:21.994037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:21.995369 (+ 1332us) server_negotiation.cc:407] Connection header received
0927 13:54:21.997714 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:21.997718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:21.997796 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:21.997873 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:21.999406 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.000134 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.001718 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.002106 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.002220 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.005678 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:22.005683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:22.005686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:22.005720 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:22.008546 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:22.009350 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:22.009354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:22.009357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:22.009490 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:22.009950 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:22.009954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:22.009956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:22.010250 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:22.010367 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:22.011051 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 13:54:22.011238 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:54:22.012595 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:21.994310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:39569 (local address 127.20.133.67:52825)
0927 13:54:21.995175 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:21.995246 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:21.997432 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:21.998083 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:21.998093 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:21.998663 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:21.999251 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:21.999268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.000277 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.000281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.001550 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:22.001561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.002319 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.002324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.002541 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.003184 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:54:22.003220 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:22.005445 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:22.008711 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:22.008719 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:22.008734 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:22.009195 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:22.009617 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:22.009620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:22.009623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:22.009832 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:22.010381 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:22.010387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:22.010747 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 13:54:22.010976 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 13:54:22.011390 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":326,"threads_started":1}
I20260927 13:54:22.013995 7265 heartbeater.cc:347] Connected to a master server at 127.20.133.126:39569
I20260927 13:54:22.014300 7265 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:22.015033 7265 heartbeater.cc:511] Master 127.20.133.126:39569 requested a full tablet report, sending...
I20260927 13:54:22.016625 6808 ts_manager.cc:194] Registered new tserver with Master: a7c37e6012494755bd560053e2702cc4 (127.20.133.67:44275)
I20260927 13:54:22.017426 6808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:52825
I20260927 13:54:22.021703 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:54:22.037961 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.025728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.20.133.126:39569)
0927 13:54:22.025876 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:22.025880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:22.026312 (+ 432us) server_negotiation.cc:407] Connection header received
0927 13:54:22.026469 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:22.026472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:22.026532 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:22.026615 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:22.028016 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.028797 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.030025 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.030449 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.030542 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.032958 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:22.032963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:22.032966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:22.032997 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:22.035491 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:22.036138 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:22.036141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:22.036144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:22.036268 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:22.036644 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:22.036647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:22.036649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:22.036932 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:22.037130 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:22.037534 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 13:54:22.037676 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 13:54:22.041832 6808 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:54:22.048804 6808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:54:22.073189 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.061712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:39405 (local address 127.0.0.1:38060)
0927 13:54:22.063855 (+ 2143us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:22.063869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:22.063981 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:22.064873 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:22.064878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:22.064911 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:22.065376 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:22.065387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.067490 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.067496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.070060 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:22.070073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.072034 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.072040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.072264 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.072315 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 13:54:22.072490 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 13:54:22.072706 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2038,"mutex_wait_us":98,"thread_start_us":1923,"threads_started":1}
I20260927 13:54:22.073572 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.061661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.20.133.66:44677)
0927 13:54:22.063748 (+ 2087us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:22.063757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:22.063786 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:54:22.063892 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:22.063900 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:22.064151 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:22.065222 (+ 1071us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:22.065450 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.067134 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.070667 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.072117 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.072254 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.072467 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:22.073078 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 13:54:22.073268 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1934,"thread_start_us":1808,"threads_started":1}
I20260927 13:54:22.073931 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.062285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.20.133.65:39405)
0927 13:54:22.063483 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:22.063493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:22.063963 (+ 470us) server_negotiation.cc:407] Connection header received
0927 13:54:22.064140 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:22.064148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:22.064383 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:22.064580 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:22.065598 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.067302 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.070268 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.071374 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.071505 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.071752 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:22.073310 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0927 13:54:22.073495 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":849,"threads_started":1}
I20260927 13:54:22.074301 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.061463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:44677 (local address 127.0.0.1:41926)
0927 13:54:22.062560 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:22.062612 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:22.062809 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:22.064428 (+ 1619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:22.064436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:22.064473 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:22.065004 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:22.065016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.067307 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.067314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.070490 (+ 3176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:22.070502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.072621 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.072629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.072829 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.072866 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 13:54:22.073009 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 13:54:22.073150 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":410,"thread_start_us":1015,"threads_started":1}
I20260927 13:54:22.077085 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.064229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:44275 (local address 127.0.0.1:39334)
0927 13:54:22.069164 (+ 4935us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:22.069183 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:22.069306 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:22.070055 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:22.070060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:22.070087 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:22.070499 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:22.070510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.072522 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.072527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.074579 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:22.074595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.076483 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.076489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.076667 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.076704 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 13:54:22.076764 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 13:54:22.076859 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4800,"thread_start_us":4041,"threads_started":1}
I20260927 13:54:22.077867 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.064361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.20.133.67:44275)
0927 13:54:22.068851 (+ 4490us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:22.068860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:22.069446 (+ 586us) server_negotiation.cc:407] Connection header received
0927 13:54:22.069575 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:22.069583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:22.069841 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:22.070042 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:22.070696 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.072284 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.074752 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.076089 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.076213 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.077442 (+ 1229us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:22.077503 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 13:54:22.077621 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4323,"thread_start_us":4164,"threads_started":1}
I20260927 13:54:22.077903 6932 tablet_service.cc:1467] Processing CreateTablet for tablet 968618691c93491fa810968a9ac1ea26 (DEFAULT_TABLE table=test-table [id=87142670cfe44e8aa83b033aa6f79b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:22.077906 7066 tablet_service.cc:1467] Processing CreateTablet for tablet 968618691c93491fa810968a9ac1ea26 (DEFAULT_TABLE table=test-table [id=87142670cfe44e8aa83b033aa6f79b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:22.079427 6932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968618691c93491fa810968a9ac1ea26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:22.079433 7066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968618691c93491fa810968a9ac1ea26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:22.081439 7200 tablet_service.cc:1467] Processing CreateTablet for tablet 968618691c93491fa810968a9ac1ea26 (DEFAULT_TABLE table=test-table [id=87142670cfe44e8aa83b033aa6f79b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:22.082733 7200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968618691c93491fa810968a9ac1ea26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:22.094551 7283 tablet_bootstrap.cc:492] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: Bootstrap starting.
I20260927 13:54:22.095275 7284 tablet_bootstrap.cc:492] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: Bootstrap starting.
I20260927 13:54:22.096210 7285 tablet_bootstrap.cc:492] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: Bootstrap starting.
I20260927 13:54:22.098737 7284 tablet_bootstrap.cc:654] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:22.098969 7283 tablet_bootstrap.cc:654] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:22.099488 7285 tablet_bootstrap.cc:654] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:22.100382 7283 log.cc:824] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:22.100510 7284 log.cc:824] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:22.100644 7285 log.cc:824] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:22.103714 7284 tablet_bootstrap.cc:492] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: No bootstrap required, opened a new log
I20260927 13:54:22.103717 7283 tablet_bootstrap.cc:492] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: No bootstrap required, opened a new log
I20260927 13:54:22.104009 7283 ts_tablet_manager.cc:1397] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 13:54:22.104007 7284 ts_tablet_manager.cc:1397] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 13:54:22.104270 7285 tablet_bootstrap.cc:492] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: No bootstrap required, opened a new log
I20260927 13:54:22.104514 7285 ts_tablet_manager.cc:1397] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 13:54:22.111815 7285 raft_consensus.cc:348] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b87baa4b872462aa67df88f51e4286e" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44677 } } peers { permanent_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 39405 } } peers { permanent_uuid: "a7c37e6012494755bd560053e2702cc4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44275 } }
I20260927 13:54:22.112222 7285 raft_consensus.cc:374] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:22.112353 7285 raft_consensus.cc:729] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c37e6012494755bd560053e2702cc4, State: Initialized, Role: FOLLOWER
I20260927 13:54:22.112987 7285 consensus_queue.cc:260] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b87baa4b872462aa67df88f51e4286e" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44677 } } peers { permanent_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 39405 } } peers { permanent_uuid: "a7c37e6012494755bd560053e2702cc4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44275 } }
I20260927 13:54:22.114133 7265 heartbeater.cc:503] Master 127.20.133.126:39569 was elected leader, sending a full tablet report...
I20260927 13:54:22.114530 7284 raft_consensus.cc:348] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b87baa4b872462aa67df88f51e4286e" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44677 } } peers { permanent_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 39405 } } peers { permanent_uuid: "a7c37e6012494755bd560053e2702cc4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44275 } }
I20260927 13:54:22.114895 7285 ts_tablet_manager.cc:1428] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 13:54:22.114995 7284 raft_consensus.cc:374] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:22.115082 7284 raft_consensus.cc:729] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8591a98ab7fa46bf82e8fb6ce1a84e88, State: Initialized, Role: FOLLOWER
I20260927 13:54:22.114902 7283 raft_consensus.cc:348] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b87baa4b872462aa67df88f51e4286e" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44677 } } peers { permanent_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 39405 } } peers { permanent_uuid: "a7c37e6012494755bd560053e2702cc4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44275 } }
I20260927 13:54:22.115411 7283 raft_consensus.cc:374] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:22.115551 7283 raft_consensus.cc:729] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b87baa4b872462aa67df88f51e4286e, State: Initialized, Role: FOLLOWER
I20260927 13:54:22.115823 7284 consensus_queue.cc:260] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b87baa4b872462aa67df88f51e4286e" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44677 } } peers { permanent_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 39405 } } peers { permanent_uuid: "a7c37e6012494755bd560053e2702cc4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44275 } }
I20260927 13:54:22.116228 7283 consensus_queue.cc:260] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b87baa4b872462aa67df88f51e4286e" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44677 } } peers { permanent_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 39405 } } peers { permanent_uuid: "a7c37e6012494755bd560053e2702cc4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44275 } }
I20260927 13:54:22.117671 7284 ts_tablet_manager.cc:1428] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: Time spent starting tablet: real 0.014s user 0.004s sys 0.011s
I20260927 13:54:22.124681 7131 heartbeater.cc:503] Master 127.20.133.126:39569 was elected leader, sending a full tablet report...
I20260927 13:54:22.125531 7283 ts_tablet_manager.cc:1428] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: Time spent starting tablet: real 0.021s user 0.013s sys 0.008s
I20260927 13:54:22.178810 7291 raft_consensus.cc:482] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:54:22.179397 7291 raft_consensus.cc:504] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b87baa4b872462aa67df88f51e4286e" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44677 } } peers { permanent_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 39405 } } peers { permanent_uuid: "a7c37e6012494755bd560053e2702cc4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44275 } }
I20260927 13:54:22.181203 7291 leader_election.cc:288] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8591a98ab7fa46bf82e8fb6ce1a84e88 (127.20.133.65:39405), a7c37e6012494755bd560053e2702cc4 (127.20.133.67:44275)
I20260927 13:54:22.189720 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.181746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:42239 (local address 127.20.133.65:39405)
0927 13:54:22.181936 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:22.181941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:22.182520 (+ 579us) server_negotiation.cc:407] Connection header received
0927 13:54:22.182779 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:22.182784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:22.182868 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:22.182987 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:22.183790 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.184864 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.187378 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.188639 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.188809 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.188886 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:22.189314 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 13:54:22.189445 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:54:22.189702 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.182716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:55425 (local address 127.20.133.67:44275)
0927 13:54:22.182968 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:22.182974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:22.183366 (+ 392us) server_negotiation.cc:407] Connection header received
0927 13:54:22.183555 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:22.183559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:22.183624 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:22.183704 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:22.184180 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.185176 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.187437 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.188646 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.188809 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.188886 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:22.189313 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 13:54:22.189445 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:54:22.189947 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.181652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:39405 (local address 127.20.133.66:42239)
0927 13:54:22.182386 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:22.182404 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:22.182553 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:22.183170 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:22.183175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:22.183217 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:22.183637 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:22.183647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.184992 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.184996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.187223 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:22.187239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.188876 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.188882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.189113 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.189185 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 13:54:22.189279 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 13:54:22.189455 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":44,"spinlock_wait_cycles":23936,"thread_start_us":377,"threads_started":1}
I20260927 13:54:22.189970 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.182509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:44275 (local address 127.20.133.66:55425)
0927 13:54:22.183242 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:22.183259 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:22.183400 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:22.183742 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:22.183746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:22.183773 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:22.184033 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:22.184040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.185298 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.185306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.187304 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:22.187315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.188888 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.188893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:22.189136 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.189188 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 13:54:22.189279 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 13:54:22.189455 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":105,"thread_start_us":375,"threads_started":1}
I20260927 13:54:22.191054 7220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968618691c93491fa810968a9ac1ea26" candidate_uuid: "9b87baa4b872462aa67df88f51e4286e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c37e6012494755bd560053e2702cc4" is_pre_election: true
I20260927 13:54:22.191054 6952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968618691c93491fa810968a9ac1ea26" candidate_uuid: "9b87baa4b872462aa67df88f51e4286e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" is_pre_election: true
I20260927 13:54:22.191648 6952 raft_consensus.cc:2454] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b87baa4b872462aa67df88f51e4286e in term 0.
I20260927 13:54:22.191649 7220 raft_consensus.cc:2454] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b87baa4b872462aa67df88f51e4286e in term 0.
I20260927 13:54:22.192353 7019 leader_election.cc:302] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8591a98ab7fa46bf82e8fb6ce1a84e88, 9b87baa4b872462aa67df88f51e4286e; no voters:
I20260927 13:54:22.192734 7291 raft_consensus.cc:2781] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:54:22.192875 7291 raft_consensus.cc:482] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:54:22.192982 7291 raft_consensus.cc:3037] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:22.195480 7291 raft_consensus.cc:504] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b87baa4b872462aa67df88f51e4286e" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44677 } } peers { permanent_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 39405 } } peers { permanent_uuid: "a7c37e6012494755bd560053e2702cc4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44275 } }
W20260927 13:54:22.195887 6998 tablet.cc:2369] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:54:22.196094 7291 leader_election.cc:288] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [CANDIDATE]: Term 1 election: Requested vote from peers 8591a98ab7fa46bf82e8fb6ce1a84e88 (127.20.133.65:39405), a7c37e6012494755bd560053e2702cc4 (127.20.133.67:44275)
I20260927 13:54:22.196579 6952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968618691c93491fa810968a9ac1ea26" candidate_uuid: "9b87baa4b872462aa67df88f51e4286e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88"
I20260927 13:54:22.196689 7220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968618691c93491fa810968a9ac1ea26" candidate_uuid: "9b87baa4b872462aa67df88f51e4286e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c37e6012494755bd560053e2702cc4"
I20260927 13:54:22.196797 6952 raft_consensus.cc:3037] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:22.196897 7220 raft_consensus.cc:3037] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:22.198982 6952 raft_consensus.cc:2454] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b87baa4b872462aa67df88f51e4286e in term 1.
I20260927 13:54:22.199050 7220 raft_consensus.cc:2454] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b87baa4b872462aa67df88f51e4286e in term 1.
I20260927 13:54:22.199343 7019 leader_election.cc:302] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8591a98ab7fa46bf82e8fb6ce1a84e88, 9b87baa4b872462aa67df88f51e4286e; no voters:
I20260927 13:54:22.199625 7291 raft_consensus.cc:2781] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:22.200078 7291 raft_consensus.cc:686] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 1 LEADER]: Becoming Leader. State: Replica: 9b87baa4b872462aa67df88f51e4286e, State: Running, Role: LEADER
I20260927 13:54:22.200619 7291 consensus_queue.cc:237] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b87baa4b872462aa67df88f51e4286e" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44677 } } peers { permanent_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 39405 } } peers { permanent_uuid: "a7c37e6012494755bd560053e2702cc4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44275 } }
I20260927 13:54:22.205299 6807 catalog_manager.cc:5495] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b87baa4b872462aa67df88f51e4286e (127.20.133.66). New cstate: current_term: 1 leader_uuid: "9b87baa4b872462aa67df88f51e4286e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b87baa4b872462aa67df88f51e4286e" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 44677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 39405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7c37e6012494755bd560053e2702cc4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44275 } health_report { overall_health: UNKNOWN } } }
I20260927 13:54:22.237596 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.230997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.20.133.66:44677)
0927 13:54:22.231189 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:22.231195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:22.231271 (+ 76us) server_negotiation.cc:407] Connection header received
0927 13:54:22.231464 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:22.231468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:22.231556 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:22.231680 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:54:22.232108 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.232934 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.235006 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.235447 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.235565 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.236209 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:54:22.236872 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:54:22.237034 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:22.237202 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 13:54:22.237254 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 13:54:22.242375 7132 tablet.cc:2369] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:54:22.243343 7266 tablet.cc:2369] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:54:22.250594 7220 raft_consensus.cc:1261] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4 [term 1 FOLLOWER]: Refusing update from remote peer 9b87baa4b872462aa67df88f51e4286e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:22.250607 6952 raft_consensus.cc:1261] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88 [term 1 FOLLOWER]: Refusing update from remote peer 9b87baa4b872462aa67df88f51e4286e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:22.251552 7294 consensus_queue.cc:1035] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8591a98ab7fa46bf82e8fb6ce1a84e88" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 39405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:22.251947 7291 consensus_queue.cc:1035] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c37e6012494755bd560053e2702cc4" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 44275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:22.278076 6808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:22.278409 6808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:22.282941 6808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f3b15feeb284c188229feebbc960ea0: Sending DeleteTablet for 3 replicas of tablet 968618691c93491fa810968a9ac1ea26
I20260927 13:54:22.284219 7066 tablet_service.cc:1514] Processing DeleteTablet for tablet 968618691c93491fa810968a9ac1ea26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:22 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:41926
I20260927 13:54:22.285691 7200 tablet_service.cc:1514] Processing DeleteTablet for tablet 968618691c93491fa810968a9ac1ea26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:22 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:39334
I20260927 13:54:22.286031 6932 tablet_service.cc:1514] Processing DeleteTablet for tablet 968618691c93491fa810968a9ac1ea26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:22 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:38060
I20260927 13:54:22.290202 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 6869
I20260927 13:54:22.300063 7306 tablet_replica.cc:330] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: stopping tablet replica
I20260927 13:54:22.300690 7306 raft_consensus.cc:2229] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:54:22.301579 7306 raft_consensus.cc:2258] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:22.304365 7306 ts_tablet_manager.cc:1905] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:22.312501 7306 ts_tablet_manager.cc:1918] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:22.312731 7306 log.cc:1196] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/968618691c93491fa810968a9ac1ea26
I20260927 13:54:22.313303 7306 ts_tablet_manager.cc:1939] T 968618691c93491fa810968a9ac1ea26 P 9b87baa4b872462aa67df88f51e4286e: Deleting consensus metadata
I20260927 13:54:22.315446 6795 catalog_manager.cc:4841] TS 9b87baa4b872462aa67df88f51e4286e (127.20.133.66:44677): tablet 968618691c93491fa810968a9ac1ea26 (table test-table [id=87142670cfe44e8aa83b033aa6f79b72]) successfully deleted
I20260927 13:54:22.317289 7308 tablet_replica.cc:330] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: stopping tablet replica
I20260927 13:54:22.317819 7308 raft_consensus.cc:2229] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:22.318384 7308 raft_consensus.cc:2258] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:22.319437 7307 tablet_replica.cc:330] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: stopping tablet replica
I20260927 13:54:22.319867 7307 raft_consensus.cc:2229] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:22.320360 7307 raft_consensus.cc:2258] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:22.321135 7308 ts_tablet_manager.cc:1905] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:22.322505 7307 ts_tablet_manager.cc:1905] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 13:54:22 dist-test-slave-q25r krb5kdc[6763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:54:22.328459 7308 ts_tablet_manager.cc:1918] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:22.328681 7308 log.cc:1196] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/968618691c93491fa810968a9ac1ea26
I20260927 13:54:22.329243 7308 ts_tablet_manager.cc:1939] T 968618691c93491fa810968a9ac1ea26 P 8591a98ab7fa46bf82e8fb6ce1a84e88: Deleting consensus metadata
I20260927 13:54:22.329270 7307 ts_tablet_manager.cc:1918] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:22.329432 7307 log.cc:1196] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/968618691c93491fa810968a9ac1ea26
I20260927 13:54:22.329913 7307 ts_tablet_manager.cc:1939] T 968618691c93491fa810968a9ac1ea26 P a7c37e6012494755bd560053e2702cc4: Deleting consensus metadata
I20260927 13:54:22.331629 6792 catalog_manager.cc:4841] TS a7c37e6012494755bd560053e2702cc4 (127.20.133.67:44275): tablet 968618691c93491fa810968a9ac1ea26 (table test-table [id=87142670cfe44e8aa83b033aa6f79b72]) successfully deleted
I20260927 13:54:22.331923 6792 catalog_manager.cc:4841] TS 8591a98ab7fa46bf82e8fb6ce1a84e88 (127.20.133.65:39405): tablet 968618691c93491fa810968a9ac1ea26 (table test-table [id=87142670cfe44e8aa83b033aa6f79b72]) successfully deleted
I20260927 13:54:22.335893 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.294622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.20.133.65:39405)
0927 13:54:22.294814 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:22.294821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:22.316596 (+ 21775us) server_negotiation.cc:407] Connection header received
0927 13:54:22.316846 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:22.316852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:22.316966 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:22.317091 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:22.319307 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.320440 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.322387 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.323286 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.323395 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.329141 (+ 5746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:22.329157 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:22.329175 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:22.329221 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:22.332935 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:22.333847 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:22.333854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:22.333856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:22.334013 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:22.334436 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:22.334440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:22.334442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:22.334822 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:22.335050 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:22.335415 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 13:54:22.335607 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:54:22.336534 6992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:22.427821 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 6869
I20260927 13:54:22.443006 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 7003
Sep 27 13:54:22 dist-test-slave-q25r krb5kdc[6763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:54:22.460170 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.443755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.20.133.66:44677)
0927 13:54:22.443951 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:22.443957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:22.443979 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:54:22.444112 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:22.444118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:22.444218 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:22.444358 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:22.445668 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.446890 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.448425 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.449105 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.449248 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.453905 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:22.453920 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:22.453939 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:22.453980 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:22.457196 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:22.458149 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:22.458156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:22.458159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:22.458312 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:22.458731 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:22.458736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:22.458738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:22.459134 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:22.459404 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:22.459693 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 13:54:22.459870 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:54:22.460978 7126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:22.576412 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 7003
I20260927 13:54:22.592679 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 7137
Sep 27 13:54:22 dist-test-slave-q25r krb5kdc[6763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:54:22.607784 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:22.593408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.20.133.67:44275)
0927 13:54:22.593592 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:22.593598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:22.593617 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:54:22.593816 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:22.593821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:22.593905 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:22.593990 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:22.595229 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.595959 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.597357 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:22.597767 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:22.597875 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:22.602355 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:22.602373 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:22.602390 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:22.602429 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:22.605042 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:22.605741 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:22.605749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:22.605752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:22.605895 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:22.606250 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:22.606255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:22.606257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:22.606614 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:22.606902 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:22.607231 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 13:54:22.607467 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:54:22.608494 7260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:22.696455 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 7137
I20260927 13:54:22.709728 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 6776
I20260927 13:54:22.710816 6838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:22.820895 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 6776
2026-09-27T13:54:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4365 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T13:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:22Z Disabled control of system clock
I20260927 13:54:22.894503 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:36645
--webserver_interface=127.20.133.126
--webserver_port=0
--builtin_ntp_servers=127.20.133.84:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:36645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:54:23.233389 7317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:23.233737 7317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:23.233825 7317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:23.241430 7317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:23.241577 7317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:23.241647 7317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:23.241725 7317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:23.241786 7317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:23.251479 7317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:36645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:36645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:23.252861 7317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:23.254679 7317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:23.269439 7322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:23.269378 7325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:23.269351 7323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:23.270334 7317 server_base.cc:1034] running on GCE node
I20260927 13:54:23.271230 7317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:23.272917 7317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:23.274183 7317 hybrid_clock.cc:648] HybridClock initialized: now 1790517263274115 us; error 76 us; skew 500 ppm
I20260927 13:54:23.274617 7317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:23.277309 7317 webserver.cc:460] Webserver started at http://127.20.133.126:46571/ using document root <none> and password file <none>
I20260927 13:54:23.278077 7317 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:23.278203 7317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:23.278524 7317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:23.281658 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "ef1c766a292f438db048686e9fe4cc26"
format_stamp: "Formatted at 2026-09-27 13:54:23 on dist-test-slave-q25r"
server_key: "b1251d8e5f230055facc7bd1c8cbe19e"
server_key_iv: "38540f6e53298628bef307e5a9810acb"
server_key_version: "encryptionkey@0"
I20260927 13:54:23.282490 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "ef1c766a292f438db048686e9fe4cc26"
format_stamp: "Formatted at 2026-09-27 13:54:23 on dist-test-slave-q25r"
server_key: "b1251d8e5f230055facc7bd1c8cbe19e"
server_key_iv: "38540f6e53298628bef307e5a9810acb"
server_key_version: "encryptionkey@0"
I20260927 13:54:23.288256 7317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 13:54:23.292307 7331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:23.294140 7317 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:54:23.294361 7317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "ef1c766a292f438db048686e9fe4cc26"
format_stamp: "Formatted at 2026-09-27 13:54:23 on dist-test-slave-q25r"
server_key: "b1251d8e5f230055facc7bd1c8cbe19e"
server_key_iv: "38540f6e53298628bef307e5a9810acb"
server_key_version: "encryptionkey@0"
I20260927 13:54:23.294602 7317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:23.315363 7317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:23.316227 7317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:23.316529 7317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:23.335983 7317 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:36645
I20260927 13:54:23.336038 7382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:36645 every 8 connection(s)
I20260927 13:54:23.337433 7317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:23.342417 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:23.343997 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 7317
I20260927 13:54:23.344182 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:23.344668 21013 external_mini_cluster.cc:1428] Setting key 9b0f37a475092a7fd0e651fbe2e1cbb4
I20260927 13:54:23.359233 7383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26: Bootstrap starting.
I20260927 13:54:23.362043 7386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:23.346869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.20.133.126:36645)
0927 13:54:23.348174 (+ 1305us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:23.348190 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:23.348247 (+ 57us) server_negotiation.cc:407] Connection header received
0927 13:54:23.350161 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:23.350189 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:23.350796 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:23.351191 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:23.352009 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:23.353414 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:23.355571 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:23.356535 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:23.356734 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:23.358764 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:23.358781 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:23.358815 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:23.359802 (+ 987us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:23.359945 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:23.359979 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:23.360103 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:23.360447 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 13:54:23.360877 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":565,"threads_started":1}
I20260927 13:54:23.363788 7383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:23.365391 7383 log.cc:824] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:23.368777 7383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26: No bootstrap required, opened a new log
I20260927 13:54:23.377168 7383 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c766a292f438db048686e9fe4cc26" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36645 } }
I20260927 13:54:23.377766 7383 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:23.377895 7383 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef1c766a292f438db048686e9fe4cc26, State: Initialized, Role: FOLLOWER
I20260927 13:54:23.378679 7383 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c766a292f438db048686e9fe4cc26" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36645 } }
I20260927 13:54:23.378980 7383 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:23.379144 7383 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:23.379336 7383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:23.382201 7383 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c766a292f438db048686e9fe4cc26" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36645 } }
I20260927 13:54:23.382977 7383 leader_election.cc:302] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef1c766a292f438db048686e9fe4cc26; no voters:
I20260927 13:54:23.383787 7383 leader_election.cc:288] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:23.384032 7388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:23.385408 7388 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [term 1 LEADER]: Becoming Leader. State: Replica: ef1c766a292f438db048686e9fe4cc26, State: Running, Role: LEADER
I20260927 13:54:23.386240 7388 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c766a292f438db048686e9fe4cc26" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36645 } }
I20260927 13:54:23.386760 7383 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:23.389348 7389 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef1c766a292f438db048686e9fe4cc26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c766a292f438db048686e9fe4cc26" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36645 } } }
I20260927 13:54:23.389664 7390 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef1c766a292f438db048686e9fe4cc26. Latest consensus state: current_term: 1 leader_uuid: "ef1c766a292f438db048686e9fe4cc26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c766a292f438db048686e9fe4cc26" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 36645 } } }
I20260927 13:54:23.390092 7389 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:23.390128 7390 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:23.391162 7395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:23.397918 7395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:23.408703 7395 catalog_manager.cc:1347] Generated new cluster ID: 231dc848bcae4a9babd5c7960891cf3b
I20260927 13:54:23.408852 7395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:23.432384 7395 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:23.434264 7395 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:23.446441 7395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26: Generated new TSK 0
I20260927 13:54:23.447595 7395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:23.455469 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:36645
--builtin_ntp_servers=127.20.133.84:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:54:23.853565 7407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:23.853935 7407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:23.854040 7407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:23.862543 7407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:23.862696 7407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:23.862850 7407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:54:23.874394 7407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:36645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:23.876142 7407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:23.878409 7407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:23.893710 7412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:23.893702 7415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:23.893939 7413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:23.894698 7407 server_base.cc:1034] running on GCE node
I20260927 13:54:23.895398 7407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:23.896585 7407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:23.897805 7407 hybrid_clock.cc:648] HybridClock initialized: now 1790517263897737 us; error 59 us; skew 500 ppm
I20260927 13:54:23.898185 7407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:23.900754 7407 webserver.cc:460] Webserver started at http://127.20.133.65:41273/ using document root <none> and password file <none>
I20260927 13:54:23.901533 7407 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:23.901644 7407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:23.901961 7407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:23.905110 7407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "f81bcff8dc9c4da4a41c2caa483ab021"
format_stamp: "Formatted at 2026-09-27 13:54:23 on dist-test-slave-q25r"
server_key: "2059ca853771169d0bfc5d98ad394aa5"
server_key_iv: "7ae39f7832fbf9a15081cb09a7a66e5c"
server_key_version: "encryptionkey@0"
I20260927 13:54:23.905921 7407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "f81bcff8dc9c4da4a41c2caa483ab021"
format_stamp: "Formatted at 2026-09-27 13:54:23 on dist-test-slave-q25r"
server_key: "2059ca853771169d0bfc5d98ad394aa5"
server_key_iv: "7ae39f7832fbf9a15081cb09a7a66e5c"
server_key_version: "encryptionkey@0"
I20260927 13:54:23.911967 7407 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 13:54:23.916093 7421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:23.917970 7407 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260927 13:54:23.918188 7407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "f81bcff8dc9c4da4a41c2caa483ab021"
format_stamp: "Formatted at 2026-09-27 13:54:23 on dist-test-slave-q25r"
server_key: "2059ca853771169d0bfc5d98ad394aa5"
server_key_iv: "7ae39f7832fbf9a15081cb09a7a66e5c"
server_key_version: "encryptionkey@0"
I20260927 13:54:23.918430 7407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:23.941216 7407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:23.942096 7407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:23.942382 7407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:23.943509 7407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:23.945168 7407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:23.945266 7407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:23.945372 7407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:23.945448 7407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:23.982002 7407 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:38403
I20260927 13:54:23.982151 7533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:38403 every 8 connection(s)
I20260927 13:54:23.983400 7407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:54:23.990880 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 7407
I20260927 13:54:23.991067 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:54:23.991530 21013 external_mini_cluster.cc:1428] Setting key 0a73e0af1d5b3cb721d677b28713608f
I20260927 13:54:23.994733 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:36645
--builtin_ntp_servers=127.20.133.84:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:54:23.998674 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:23.986598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:38471 (local address 127.20.133.126:36645)
0927 13:54:23.987167 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:23.987173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:23.988501 (+ 1328us) server_negotiation.cc:407] Connection header received
0927 13:54:23.990674 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:23.990680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:23.990782 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:23.990899 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:23.992524 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:23.993411 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:23.995277 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:23.995761 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:23.995878 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:23.997634 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:23.997639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:23.997642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:23.997663 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:23.997710 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:23.997723 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:23.997804 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:23.998067 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 13:54:23.998193 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":328,"threads_started":1}
I20260927 13:54:23.999863 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:23.987112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:36645 (local address 127.20.133.65:38471)
0927 13:54:23.988269 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:23.988316 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:23.990340 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:23.991198 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:23.991212 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:23.991338 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:23.992197 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:23.992225 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:23.993654 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:23.993658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:23.995129 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:23.995142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:23.996008 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:23.996015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:23.996274 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:23.997185 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:54:23.997188 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:23.997240 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:54:23.997415 (+ 175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:23.997816 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:23.997828 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:23.997837 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 13:54:23.998014 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 13:54:23.998549 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":310,"threads_started":1}
I20260927 13:54:24.001747 7534 heartbeater.cc:347] Connected to a master server at 127.20.133.126:36645
I20260927 13:54:24.002111 7534 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:24.003059 7534 heartbeater.cc:511] Master 127.20.133.126:36645 requested a full tablet report, sending...
I20260927 13:54:24.005676 7348 ts_manager.cc:194] Registered new tserver with Master: f81bcff8dc9c4da4a41c2caa483ab021 (127.20.133.65:38403)
I20260927 13:54:24.007448 7348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.65:38471
W20260927 13:54:24.349720 7538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:24.350087 7538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:24.350214 7538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:24.357729 7538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:24.357861 7538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:24.358006 7538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:54:24.367364 7538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:36645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:24.368780 7538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:24.370659 7538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:24.384163 7543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:24.384128 7546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:24.384097 7544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:24.384866 7538 server_base.cc:1034] running on GCE node
I20260927 13:54:24.385569 7538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:24.386611 7538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:24.387815 7538 hybrid_clock.cc:648] HybridClock initialized: now 1790517264387752 us; error 57 us; skew 500 ppm
I20260927 13:54:24.388151 7538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:24.390542 7538 webserver.cc:460] Webserver started at http://127.20.133.66:35529/ using document root <none> and password file <none>
I20260927 13:54:24.391247 7538 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:24.391350 7538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:24.391662 7538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:24.394731 7538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "ad2541a7175d4a0aa752a4e8e8a590e7"
format_stamp: "Formatted at 2026-09-27 13:54:24 on dist-test-slave-q25r"
server_key: "0af8f7fc3ce9c1ff75797d810520ada8"
server_key_iv: "a9803c0f2e2091182aa4e49e4975eb0a"
server_key_version: "encryptionkey@0"
I20260927 13:54:24.395581 7538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "ad2541a7175d4a0aa752a4e8e8a590e7"
format_stamp: "Formatted at 2026-09-27 13:54:24 on dist-test-slave-q25r"
server_key: "0af8f7fc3ce9c1ff75797d810520ada8"
server_key_iv: "a9803c0f2e2091182aa4e49e4975eb0a"
server_key_version: "encryptionkey@0"
I20260927 13:54:24.401013 7538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260927 13:54:24.404755 7552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:24.406181 7538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:54:24.406356 7538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "ad2541a7175d4a0aa752a4e8e8a590e7"
format_stamp: "Formatted at 2026-09-27 13:54:24 on dist-test-slave-q25r"
server_key: "0af8f7fc3ce9c1ff75797d810520ada8"
server_key_iv: "a9803c0f2e2091182aa4e49e4975eb0a"
server_key_version: "encryptionkey@0"
I20260927 13:54:24.406585 7538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:24.421095 7538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:24.422039 7538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:24.422339 7538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:24.423418 7538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:24.425031 7538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:24.425138 7538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:24.425258 7538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:24.425328 7538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:24.457269 7538 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:38849
I20260927 13:54:24.457289 7664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:38849 every 8 connection(s)
I20260927 13:54:24.458673 7538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:54:24.461490 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 7538
I20260927 13:54:24.461673 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:54:24.462097 21013 external_mini_cluster.cc:1428] Setting key 20d2ddd616c3ebd55f5357ab2f0a8782
I20260927 13:54:24.465301 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:36645
--builtin_ntp_servers=127.20.133.84:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:54:24.472476 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:24.461348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:50253 (local address 127.20.133.126:36645)
0927 13:54:24.461583 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:24.461590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:24.463069 (+ 1479us) server_negotiation.cc:407] Connection header received
0927 13:54:24.465256 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:24.465260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:24.465331 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:24.465429 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:24.466850 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.467631 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.469668 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.470089 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.470184 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:24.471691 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:24.471695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:24.471699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:24.471720 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:24.471757 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:24.471770 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:24.471846 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:24.472062 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 13:54:24.472159 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:54:24.473816 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:24.461905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:36645 (local address 127.20.133.66:50253)
0927 13:54:24.462908 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:24.462954 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:24.464949 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:24.465712 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:24.465723 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:24.465838 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:24.466646 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:24.466671 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.467770 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.467775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:24.469518 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:24.469533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.470298 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.470304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:24.470581 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:24.471329 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:54:24.471332 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:24.471384 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:54:24.471484 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:24.471859 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:24.471868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:24.471875 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 13:54:24.472053 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 13:54:24.472498 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":292,"threads_started":1}
I20260927 13:54:24.475257 7665 heartbeater.cc:347] Connected to a master server at 127.20.133.126:36645
I20260927 13:54:24.475565 7665 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:24.476321 7665 heartbeater.cc:511] Master 127.20.133.126:36645 requested a full tablet report, sending...
I20260927 13:54:24.477970 7348 ts_manager.cc:194] Registered new tserver with Master: ad2541a7175d4a0aa752a4e8e8a590e7 (127.20.133.66:38849)
I20260927 13:54:24.478981 7348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.66:50253
W20260927 13:54:24.794445 7668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:24.794749 7668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:24.794837 7668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:24.802408 7668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:24.802520 7668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:24.802635 7668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:54:24.811966 7668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:36645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:24.813253 7668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:24.814908 7668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:24.828420 7676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:24.828377 7673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:24.828370 7674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:24.829150 7668 server_base.cc:1034] running on GCE node
I20260927 13:54:24.829782 7668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:24.830812 7668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:24.832050 7668 hybrid_clock.cc:648] HybridClock initialized: now 1790517264831990 us; error 52 us; skew 500 ppm
I20260927 13:54:24.832448 7668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:24.834743 7668 webserver.cc:460] Webserver started at http://127.20.133.67:36633/ using document root <none> and password file <none>
I20260927 13:54:24.835470 7668 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:24.835573 7668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:24.835873 7668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:24.838856 7668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "b1c254adf8e24504b656fca1c3616016"
format_stamp: "Formatted at 2026-09-27 13:54:24 on dist-test-slave-q25r"
server_key: "0daa5f61913b0f0e95701fc6def363ff"
server_key_iv: "811b7ef2fadeaa43cc58749b20bfafaf"
server_key_version: "encryptionkey@0"
I20260927 13:54:24.839633 7668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "b1c254adf8e24504b656fca1c3616016"
format_stamp: "Formatted at 2026-09-27 13:54:24 on dist-test-slave-q25r"
server_key: "0daa5f61913b0f0e95701fc6def363ff"
server_key_iv: "811b7ef2fadeaa43cc58749b20bfafaf"
server_key_version: "encryptionkey@0"
I20260927 13:54:24.844753 7668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 13:54:24.848163 7682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:24.849506 7668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:54:24.849666 7668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "b1c254adf8e24504b656fca1c3616016"
format_stamp: "Formatted at 2026-09-27 13:54:24 on dist-test-slave-q25r"
server_key: "0daa5f61913b0f0e95701fc6def363ff"
server_key_iv: "811b7ef2fadeaa43cc58749b20bfafaf"
server_key_version: "encryptionkey@0"
I20260927 13:54:24.849865 7668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:24.868084 7668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:24.869179 7668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:24.869520 7668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:24.870563 7668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:24.872020 7668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:24.872118 7668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:24.872228 7668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:24.872294 7668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:24.904057 7668 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:39251
I20260927 13:54:24.904069 7794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:39251 every 8 connection(s)
I20260927 13:54:24.905443 7668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:54:24.913374 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 7668
I20260927 13:54:24.913540 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:54:24.914098 21013 external_mini_cluster.cc:1428] Setting key 2780754bbb112524bf5a35ecf4d949d5
I20260927 13:54:24.919229 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:24.907946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:34715 (local address 127.20.133.126:36645)
0927 13:54:24.908134 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:24.908138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:24.909830 (+ 1692us) server_negotiation.cc:407] Connection header received
0927 13:54:24.911861 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:24.911867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:24.911958 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:24.912045 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:24.913333 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.914180 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.916212 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.916721 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.916822 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:24.918405 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:24.918410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:24.918413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:24.918434 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:24.918471 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:24.918484 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:24.918564 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:24.918878 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 13:54:24.918973 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 13:54:24.920289 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:24.908464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:36645 (local address 127.20.133.67:34715)
0927 13:54:24.909621 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:24.909673 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:24.911521 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:24.912326 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:24.912367 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:24.912487 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:24.913138 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:24.913157 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.914354 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.914358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:24.916073 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:24.916091 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.916981 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.916989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:24.917310 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:24.918040 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:54:24.918043 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:24.918097 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:54:24.918190 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:24.918613 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:24.918622 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:24.918630 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:54:24.918805 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 13:54:24.919154 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":300,"threads_started":1}
I20260927 13:54:24.921916 7795 heartbeater.cc:347] Connected to a master server at 127.20.133.126:36645
I20260927 13:54:24.922227 7795 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:24.923120 7795 heartbeater.cc:511] Master 127.20.133.126:36645 requested a full tablet report, sending...
I20260927 13:54:24.924815 7348 ts_manager.cc:194] Registered new tserver with Master: b1c254adf8e24504b656fca1c3616016 (127.20.133.67:39251)
I20260927 13:54:24.925587 7348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.67:34715
I20260927 13:54:24.929899 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:54:24.939762 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:24.933844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.20.133.126:36645)
0927 13:54:24.934052 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:24.934059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:24.934418 (+ 359us) server_negotiation.cc:407] Connection header received
0927 13:54:24.934589 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:24.934593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:24.934657 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:24.934737 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:24.935250 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.935950 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.937330 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.937764 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.937881 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:24.939054 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:24.939059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:24.939063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:24.939083 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:24.939126 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:24.939138 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:24.939255 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:24.939434 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 13:54:24.939491 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:54:24.943485 7348 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:54:24.950317 7348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:54:24.976226 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:24.963635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:39251 (local address 127.0.0.1:58628)
0927 13:54:24.966374 (+ 2739us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:24.966390 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:24.966492 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:24.967698 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:24.967707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:24.967736 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:24.968230 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:24.968246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.970525 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.970530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:24.973241 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:24.973262 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.975218 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.975225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:24.975415 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:24.975475 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 13:54:24.975602 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 13:54:24.975818 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2608,"mutex_wait_us":1,"thread_start_us":2473,"threads_started":1}
I20260927 13:54:24.977999 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:24.964164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.20.133.67:39251)
0927 13:54:24.967062 (+ 2898us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:24.967070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:24.967107 (+ 37us) server_negotiation.cc:407] Connection header received
0927 13:54:24.967201 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:24.967209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:24.967426 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:24.968470 (+ 1044us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:24.968766 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.970357 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.973522 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.974995 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.977023 (+ 2028us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:24.977343 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:24.977420 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 13:54:24.977621 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2727,"thread_start_us":2567,"threads_started":1}
I20260927 13:54:24.978631 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:24.963561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38403 (local address 127.0.0.1:49912)
0927 13:54:24.965719 (+ 2158us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:24.965756 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:24.965941 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:24.968510 (+ 2569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:24.968515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:24.968539 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:24.968909 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:24.968920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.972421 (+ 3501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.972427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:24.974533 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:24.974548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.977870 (+ 3322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.977876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:24.978071 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:24.978106 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:54:24.978192 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 13:54:24.978328 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1915,"thread_start_us":1754,"threads_started":1}
I20260927 13:54:24.979193 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:24.963892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.20.133.65:38403)
0927 13:54:24.965154 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:24.965166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:24.966177 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 13:54:24.966296 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:24.966303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:24.966574 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:24.968195 (+ 1621us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:24.969129 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.972228 (+ 3099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.974755 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.977668 (+ 2913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.977802 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:24.978022 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:24.978705 (+ 683us) server_negotiation.cc:299] Negotiation successful
0927 13:54:24.978878 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":986,"threads_started":1}
I20260927 13:54:24.982086 7730 tablet_service.cc:1467] Processing CreateTablet for tablet 82ae0bdace904567833ce5243825239e (DEFAULT_TABLE table=test-table [id=861d4c329a3c4b5d81c81bffa2413f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:24.982513 7469 tablet_service.cc:1467] Processing CreateTablet for tablet 82ae0bdace904567833ce5243825239e (DEFAULT_TABLE table=test-table [id=861d4c329a3c4b5d81c81bffa2413f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:24.983642 7730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ae0bdace904567833ce5243825239e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:24.983839 7469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ae0bdace904567833ce5243825239e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:24.984925 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:24.966550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:38849 (local address 127.0.0.1:43408)
0927 13:54:24.976490 (+ 9940us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:24.976510 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:24.976618 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:24.978419 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:24.978423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:24.978452 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:24.978627 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:24.978637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.980424 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.980430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:24.982375 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:24.982392 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.984174 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.984180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:24.984398 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:24.984435 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 13:54:24.984504 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 13:54:24.984651 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9839,"thread_start_us":10360,"threads_started":1}
I20260927 13:54:24.986089 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:24.967240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.20.133.66:38849)
0927 13:54:24.977778 (+ 10538us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:24.977787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:24.977816 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:54:24.977924 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:24.977933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:24.978210 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:24.978417 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:24.978792 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.980250 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.982558 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:24.983967 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:24.985298 (+ 1331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:24.985523 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:24.985604 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 13:54:24.985774 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10392,"thread_start_us":10263,"threads_started":1}
I20260927 13:54:24.992306 7600 tablet_service.cc:1467] Processing CreateTablet for tablet 82ae0bdace904567833ce5243825239e (DEFAULT_TABLE table=test-table [id=861d4c329a3c4b5d81c81bffa2413f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:24.993913 7600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ae0bdace904567833ce5243825239e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:24.998442 7813 tablet_bootstrap.cc:492] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: Bootstrap starting.
I20260927 13:54:25.000677 7814 tablet_bootstrap.cc:492] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021: Bootstrap starting.
I20260927 13:54:25.002398 7813 tablet_bootstrap.cc:654] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:25.003568 7813 log.cc:824] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:25.004197 7814 tablet_bootstrap.cc:654] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:25.005640 7814 log.cc:824] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:25.007597 7813 tablet_bootstrap.cc:492] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: No bootstrap required, opened a new log
I20260927 13:54:25.007866 7813 ts_tablet_manager.cc:1397] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260927 13:54:25.008787 7814 tablet_bootstrap.cc:492] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021: No bootstrap required, opened a new log
I20260927 13:54:25.009166 7814 ts_tablet_manager.cc:1397] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 13:54:25.009193 7817 tablet_bootstrap.cc:492] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7: Bootstrap starting.
I20260927 13:54:25.011224 7534 heartbeater.cc:503] Master 127.20.133.126:36645 was elected leader, sending a full tablet report...
I20260927 13:54:25.013128 7817 tablet_bootstrap.cc:654] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:25.014483 7817 log.cc:824] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:25.019364 7817 tablet_bootstrap.cc:492] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7: No bootstrap required, opened a new log
I20260927 13:54:25.019704 7817 ts_tablet_manager.cc:1397] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 13:54:25.019541 7814 raft_consensus.cc:348] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 } } peers { permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 } } peers { permanent_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38849 } }
I20260927 13:54:25.019456 7813 raft_consensus.cc:348] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 } } peers { permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 } } peers { permanent_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38849 } }
I20260927 13:54:25.019932 7814 raft_consensus.cc:374] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:25.019932 7813 raft_consensus.cc:374] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:25.020032 7814 raft_consensus.cc:729] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81bcff8dc9c4da4a41c2caa483ab021, State: Initialized, Role: FOLLOWER
I20260927 13:54:25.020154 7813 raft_consensus.cc:729] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c254adf8e24504b656fca1c3616016, State: Initialized, Role: FOLLOWER
I20260927 13:54:25.020804 7814 consensus_queue.cc:260] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 } } peers { permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 } } peers { permanent_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38849 } }
I20260927 13:54:25.020872 7813 consensus_queue.cc:260] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 } } peers { permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 } } peers { permanent_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38849 } }
I20260927 13:54:25.024077 7814 ts_tablet_manager.cc:1428] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021: Time spent starting tablet: real 0.015s user 0.006s sys 0.005s
I20260927 13:54:25.025076 7795 heartbeater.cc:503] Master 127.20.133.126:36645 was elected leader, sending a full tablet report...
I20260927 13:54:25.025723 7813 ts_tablet_manager.cc:1428] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: Time spent starting tablet: real 0.018s user 0.015s sys 0.002s
I20260927 13:54:25.030691 7817 raft_consensus.cc:348] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 } } peers { permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 } } peers { permanent_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38849 } }
I20260927 13:54:25.031134 7817 raft_consensus.cc:374] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:25.031240 7817 raft_consensus.cc:729] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2541a7175d4a0aa752a4e8e8a590e7, State: Initialized, Role: FOLLOWER
I20260927 13:54:25.031854 7817 consensus_queue.cc:260] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 } } peers { permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 } } peers { permanent_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38849 } }
I20260927 13:54:25.032971 7665 heartbeater.cc:503] Master 127.20.133.126:36645 was elected leader, sending a full tablet report...
I20260927 13:54:25.033483 7817 ts_tablet_manager.cc:1428] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260927 13:54:25.117563 7821 raft_consensus.cc:482] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:54:25.117914 7821 raft_consensus.cc:504] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 } } peers { permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 } } peers { permanent_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38849 } }
I20260927 13:54:25.119684 7821 leader_election.cc:288] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f81bcff8dc9c4da4a41c2caa483ab021 (127.20.133.65:38403), b1c254adf8e24504b656fca1c3616016 (127.20.133.67:39251)
I20260927 13:54:25.129966 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.120295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:40201 (local address 127.20.133.65:38403)
0927 13:54:25.120537 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:25.120544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:25.121852 (+ 1308us) server_negotiation.cc:407] Connection header received
0927 13:54:25.122183 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:25.122189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:25.122275 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:25.122376 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:25.123362 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.124500 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.127608 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.128861 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.129036 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.129117 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:25.129564 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 13:54:25.129701 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 13:54:25.129962 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.120755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:47963 (local address 127.20.133.67:39251)
0927 13:54:25.121106 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:25.121115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:25.121844 (+ 729us) server_negotiation.cc:407] Connection header received
0927 13:54:25.122184 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:25.122189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:25.122275 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:25.122376 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:25.123371 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.124500 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.127622 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.128909 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.129049 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.129117 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:25.129564 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 13:54:25.129705 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 13:54:25.130134 7819 raft_consensus.cc:482] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:54:25.130353 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.120127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38403 (local address 127.20.133.66:40201)
0927 13:54:25.121700 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:25.121723 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:25.121999 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:25.122442 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:25.122446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:25.122480 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:25.123163 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:25.123172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.124664 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.124670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:25.127322 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:25.127344 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.129076 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.129084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:25.129365 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.129433 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 13:54:25.129527 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 13:54:25.129701 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"mutex_wait_us":394,"spinlock_wait_cycles":284160,"thread_start_us":1182,"threads_started":1}
I20260927 13:54:25.130390 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.120176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:39251 (local address 127.20.133.66:47963)
0927 13:54:25.121706 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:25.121722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:25.121987 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:25.122436 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:25.122442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:25.122481 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:25.123122 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:25.123141 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.124664 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.124670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:25.127322 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:25.127344 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.129079 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.129084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:25.129364 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.129432 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 13:54:25.129528 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 13:54:25.129701 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"mutex_wait_us":239,"spinlock_wait_cycles":49408,"thread_start_us":689,"threads_started":1}
I20260927 13:54:25.130586 7819 raft_consensus.cc:504] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 } } peers { permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 } } peers { permanent_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38849 } }
I20260927 13:54:25.131392 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae0bdace904567833ce5243825239e" candidate_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1c254adf8e24504b656fca1c3616016" is_pre_election: true
I20260927 13:54:25.131649 7489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae0bdace904567833ce5243825239e" candidate_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" is_pre_election: true
I20260927 13:54:25.131846 7750 raft_consensus.cc:2454] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2541a7175d4a0aa752a4e8e8a590e7 in term 0.
I20260927 13:54:25.132061 7489 raft_consensus.cc:2454] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2541a7175d4a0aa752a4e8e8a590e7 in term 0.
I20260927 13:54:25.132544 7555 leader_election.cc:302] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2541a7175d4a0aa752a4e8e8a590e7, b1c254adf8e24504b656fca1c3616016; no voters:
I20260927 13:54:25.132819 7819 leader_election.cc:288] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1c254adf8e24504b656fca1c3616016 (127.20.133.67:39251), ad2541a7175d4a0aa752a4e8e8a590e7 (127.20.133.66:38849)
I20260927 13:54:25.133045 7821 raft_consensus.cc:2781] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:54:25.133209 7821 raft_consensus.cc:482] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:54:25.133312 7821 raft_consensus.cc:3037] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:25.136250 7821 raft_consensus.cc:504] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 } } peers { permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 } } peers { permanent_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38849 } }
I20260927 13:54:25.137061 7821 leader_election.cc:288] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [CANDIDATE]: Term 1 election: Requested vote from peers f81bcff8dc9c4da4a41c2caa483ab021 (127.20.133.65:38403), b1c254adf8e24504b656fca1c3616016 (127.20.133.67:39251)
I20260927 13:54:25.137398 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae0bdace904567833ce5243825239e" candidate_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1c254adf8e24504b656fca1c3616016"
I20260927 13:54:25.137645 7750 raft_consensus.cc:3037] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:25.137655 7489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae0bdace904567833ce5243825239e" candidate_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81bcff8dc9c4da4a41c2caa483ab021"
I20260927 13:54:25.137868 7489 raft_consensus.cc:3037] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:25.141040 7750 raft_consensus.cc:2454] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2541a7175d4a0aa752a4e8e8a590e7 in term 1.
I20260927 13:54:25.141141 7489 raft_consensus.cc:2454] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2541a7175d4a0aa752a4e8e8a590e7 in term 1.
I20260927 13:54:25.141636 7554 leader_election.cc:302] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2541a7175d4a0aa752a4e8e8a590e7, f81bcff8dc9c4da4a41c2caa483ab021; no voters:
I20260927 13:54:25.141788 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.133191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:53099 (local address 127.20.133.67:39251)
0927 13:54:25.133374 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:25.133380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:25.134265 (+ 885us) server_negotiation.cc:407] Connection header received
0927 13:54:25.134515 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:25.134520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:25.134608 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:25.134668 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:25.135409 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.136534 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.139289 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.140604 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.140753 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.140825 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:25.141363 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 13:54:25.141504 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:54:25.141996 7821 raft_consensus.cc:2781] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:25.142180 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.134348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:50425 (local address 127.20.133.66:38849)
0927 13:54:25.134545 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:25.134555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:25.135105 (+ 550us) server_negotiation.cc:407] Connection header received
0927 13:54:25.135257 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:25.135262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:25.135354 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:25.135432 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:25.136172 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.137241 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.139853 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.141060 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.141188 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.141248 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:25.141742 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 13:54:25.141895 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:54:25.142436 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.133130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:39251 (local address 127.20.133.65:53099)
0927 13:54:25.134172 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:25.134190 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:25.134373 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:25.134862 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:25.134866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:25.134892 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:25.135257 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:25.135269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.136704 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.136710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:25.139033 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:25.139052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.141004 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.141011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:25.141183 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.141218 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:54:25.141948 (+ 730us) client_negotiation.cc:240] Negotiation successful
0927 13:54:25.142099 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":65,"thread_start_us":709,"threads_started":1}
I20260927 13:54:25.142663 7821 raft_consensus.cc:686] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 1 LEADER]: Becoming Leader. State: Replica: ad2541a7175d4a0aa752a4e8e8a590e7, State: Running, Role: LEADER
I20260927 13:54:25.142768 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.134242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:38849 (local address 127.20.133.65:50425)
0927 13:54:25.134996 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:25.135018 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:25.135148 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:25.135582 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:25.135587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:25.135609 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:25.135972 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:25.135984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.137420 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.137429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:25.139641 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:25.139654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.141403 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.141409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:25.141559 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.141591 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:54:25.142331 (+ 740us) client_negotiation.cc:240] Negotiation successful
0927 13:54:25.142467 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":4,"thread_start_us":468,"threads_started":1}
I20260927 13:54:25.143368 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae0bdace904567833ce5243825239e" candidate_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1c254adf8e24504b656fca1c3616016" is_pre_election: true
I20260927 13:54:25.143405 7821 consensus_queue.cc:237] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 } } peers { permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 } } peers { permanent_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38849 } }
I20260927 13:54:25.143674 7750 raft_consensus.cc:2379] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f81bcff8dc9c4da4a41c2caa483ab021 in current term 1: Already voted for candidate ad2541a7175d4a0aa752a4e8e8a590e7 in this term.
I20260927 13:54:25.143749 7620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae0bdace904567833ce5243825239e" candidate_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" is_pre_election: true
I20260927 13:54:25.144646 7424 leader_election.cc:302] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f81bcff8dc9c4da4a41c2caa483ab021; no voters: ad2541a7175d4a0aa752a4e8e8a590e7, b1c254adf8e24504b656fca1c3616016
I20260927 13:54:25.145059 7819 raft_consensus.cc:2726] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 13:54:25.148432 7348 catalog_manager.cc:5495] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad2541a7175d4a0aa752a4e8e8a590e7 (127.20.133.66). New cstate: current_term: 1 leader_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2541a7175d4a0aa752a4e8e8a590e7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 38849 } health_report { overall_health: HEALTHY } } }
W20260927 13:54:25.157094 7796 tablet.cc:2369] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:54:25.174479 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.167870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.20.133.66:38849)
0927 13:54:25.168042 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:25.168048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:25.168067 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:54:25.168187 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:25.168193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:25.168368 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:25.168533 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:54:25.168985 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.169754 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.171726 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.172462 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.172591 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.173193 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:54:25.173852 (+ 659us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:54:25.174004 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:25.174131 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 13:54:25.174170 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:54:25.185350 7750 raft_consensus.cc:1261] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016 [term 1 FOLLOWER]: Refusing update from remote peer ad2541a7175d4a0aa752a4e8e8a590e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:25.185402 7489 raft_consensus.cc:1261] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 1 FOLLOWER]: Refusing update from remote peer ad2541a7175d4a0aa752a4e8e8a590e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:25.186391 7826 consensus_queue.cc:1035] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1c254adf8e24504b656fca1c3616016" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 39251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:25.186820 7821 consensus_queue.cc:1035] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f81bcff8dc9c4da4a41c2caa483ab021" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:25.198653 7829 mvcc.cc:205] Tried to move back new op lower bound from 7333958718187393024 to 7333958718032633856. Current Snapshot: MvccSnapshot[applied={T|T < 7333958718187393024}]
I20260927 13:54:25.214526 7348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:25.214828 7348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:25.218847 7348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef1c766a292f438db048686e9fe4cc26: Sending DeleteTablet for 3 replicas of tablet 82ae0bdace904567833ce5243825239e
I20260927 13:54:25.220080 7469 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ae0bdace904567833ce5243825239e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:25 UTC) from {username='slave'} at 127.0.0.1:49912
I20260927 13:54:25.220242 7730 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ae0bdace904567833ce5243825239e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:25 UTC) from {username='slave'} at 127.0.0.1:58628
I20260927 13:54:25.220620 7600 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ae0bdace904567833ce5243825239e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:25 UTC) from {username='slave'} at 127.0.0.1:43408
I20260927 13:54:25.225863 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 7407
I20260927 13:54:25.232937 7841 tablet_replica.cc:330] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7: stopping tablet replica
I20260927 13:54:25.233517 7841 raft_consensus.cc:2229] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:54:25.234385 7841 raft_consensus.cc:2258] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:25.237041 7841 ts_tablet_manager.cc:1905] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:25.245704 7841 ts_tablet_manager.cc:1918] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:25.246043 7841 log.cc:1196] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/82ae0bdace904567833ce5243825239e
I20260927 13:54:25.246783 7841 ts_tablet_manager.cc:1939] T 82ae0bdace904567833ce5243825239e P ad2541a7175d4a0aa752a4e8e8a590e7: Deleting consensus metadata
I20260927 13:54:25.249591 7334 catalog_manager.cc:4841] TS ad2541a7175d4a0aa752a4e8e8a590e7 (127.20.133.66:38849): tablet 82ae0bdace904567833ce5243825239e (table test-table [id=861d4c329a3c4b5d81c81bffa2413f39]) successfully deleted
I20260927 13:54:25.251873 7839 tablet_replica.cc:330] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021: stopping tablet replica
I20260927 13:54:25.252419 7839 raft_consensus.cc:2229] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:25.252976 7839 raft_consensus.cc:2258] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:25.253770 7840 tablet_replica.cc:330] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: stopping tablet replica
I20260927 13:54:25.254256 7840 raft_consensus.cc:2229] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:25.254822 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.227560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.20.133.65:38403)
0927 13:54:25.227800 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:25.227811 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:25.245565 (+ 17754us) server_negotiation.cc:407] Connection header received
0927 13:54:25.246112 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:25.246118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:25.246264 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:25.246350 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:25.247058 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.248414 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.250676 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.251417 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.251574 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.253096 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:25.253108 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:25.253124 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:25.253916 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:25.253982 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:25.254008 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:25.254099 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:25.254363 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 13:54:25.254493 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 13:54:25.254894 7840 raft_consensus.cc:2258] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:25.256310 7529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:25.257345 7840 ts_tablet_manager.cc:1905] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:25.265610 7840 ts_tablet_manager.cc:1918] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:25.265820 7840 log.cc:1196] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/82ae0bdace904567833ce5243825239e
I20260927 13:54:25.266409 7840 ts_tablet_manager.cc:1939] T 82ae0bdace904567833ce5243825239e P b1c254adf8e24504b656fca1c3616016: Deleting consensus metadata
I20260927 13:54:25.268537 7334 catalog_manager.cc:4841] TS b1c254adf8e24504b656fca1c3616016 (127.20.133.67:39251): tablet 82ae0bdace904567833ce5243825239e (table test-table [id=861d4c329a3c4b5d81c81bffa2413f39]) successfully deleted
I20260927 13:54:25.257287 7839 ts_tablet_manager.cc:1905] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:25.358698 7839 ts_tablet_manager.cc:1918] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:25.358882 7839 log.cc:1196] T 82ae0bdace904567833ce5243825239e P f81bcff8dc9c4da4a41c2caa483ab021: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/82ae0bdace904567833ce5243825239e
I20260927 13:54:25.359272 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 7407
W20260927 13:54:25.374377 7333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.133.65:38403 (error 108)
I20260927 13:54:25.375712 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 7538
I20260927 13:54:25.376268 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.375751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38403 (local address 127.0.0.1:49936)
0927 13:54:25.375975 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:25.376082 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 13:54:25.376540 7333 catalog_manager.cc:4584] TS f81bcff8dc9c4da4a41c2caa483ab021 (127.20.133.65:38403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82ae0bdace904567833ce5243825239e: Network error: Client connection negotiation failed: client connection to 127.20.133.65:38403: connect: Connection refused (error 111)
I20260927 13:54:25.383311 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.376524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.20.133.66:38849)
0927 13:54:25.376715 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:25.376725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:25.376855 (+ 130us) server_negotiation.cc:407] Connection header received
0927 13:54:25.377014 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:25.377020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:25.377135 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:25.377223 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:25.377641 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.378402 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.379669 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.380139 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.380281 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.381565 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:25.381576 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:25.381592 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:25.382214 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:25.382266 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:25.382290 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:25.382391 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:25.382604 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 13:54:25.382777 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:54:25.383988 7660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:25.430467 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.429971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38403 (local address 127.0.0.1:49950)
0927 13:54:25.430198 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:25.430290 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 13:54:25.431494 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.431154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38403 (local address 127.0.0.1:49962)
0927 13:54:25.431282 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:25.431352 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 13:54:25.492057 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.491571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38403 (local address 127.0.0.1:49976)
0927 13:54:25.491780 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:25.491894 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 13:54:25.493085 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.492733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38403 (local address 127.0.0.1:49978)
0927 13:54:25.492873 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:25.492947 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 13:54:25.497797 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 7538
I20260927 13:54:25.513671 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 7668
I20260927 13:54:25.520407 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.514343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.20.133.67:39251)
0927 13:54:25.514541 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:25.514547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:25.514567 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:54:25.514756 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:25.514761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:25.514854 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:25.514940 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:25.515357 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.516048 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.517417 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:25.517829 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:25.517949 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:25.519037 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:25.519046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:25.519067 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:25.519646 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:25.519694 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:25.519714 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:25.519812 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:25.519967 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 13:54:25.520083 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:54:25.521118 7790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:25.573923 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.573339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38403 (local address 127.0.0.1:49982)
0927 13:54:25.573612 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:25.573738 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 13:54:25.575191 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:25.574670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:38403 (local address 127.0.0.1:49986)
0927 13:54:25.574900 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:25.575003 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260927 13:54:25.611413 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 7668
I20260927 13:54:25.624886 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 7317
I20260927 13:54:25.626091 7378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:25.738380 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 7317
2026-09-27T13:54:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2917 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:25 dist-test-slave-q25r krb5kdc[7850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:25 dist-test-slave-q25r krb5kdc[7850](info): set up 2 sockets
Sep 27 13:54:25 dist-test-slave-q25r krb5kdc[7850](info): commencing operation
krb5kdc: starting...
W20260927 13:54:26.776741 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:26 dist-test-slave-q25r krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:54:27.068437 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:44805
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:27.425194 7863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:27.425516 7863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:27.425608 7863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:27.433681 7863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:27.433883 7863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:27.433993 7863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:27.434099 7863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:27.434172 7863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:27.444693 7863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:44805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:27.446134 7863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:27.447499 7863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:27.460229 7868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:27.460173 7871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:27.460248 7869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:27.461560 7863 server_base.cc:1034] running on GCE node
I20260927 13:54:27.462407 7863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:27.463937 7863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:27.465188 7863 hybrid_clock.cc:648] HybridClock initialized: now 1790517267465129 us; error 47 us; skew 500 ppm
I20260927 13:54:27.465572 7863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:27 dist-test-slave-q25r krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517267, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:27.474577 7863 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:54:27.476255 7863 webserver.cc:460] Webserver started at http://127.20.133.126:44155/ using document root <none> and password file <none>
I20260927 13:54:27.477021 7863 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:27.477166 7863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:27.477476 7863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:27.480746 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "548510bc8cf144e1970d2f13d02316cf"
format_stamp: "Formatted at 2026-09-27 13:54:27 on dist-test-slave-q25r"
server_key: "6aa15e84a73c64188b3a99caadb5136f"
server_key_iv: "18d1b589eb866f627e6ae585774b8fd2"
server_key_version: "encryptionkey@0"
I20260927 13:54:27.481643 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "548510bc8cf144e1970d2f13d02316cf"
format_stamp: "Formatted at 2026-09-27 13:54:27 on dist-test-slave-q25r"
server_key: "6aa15e84a73c64188b3a99caadb5136f"
server_key_iv: "18d1b589eb866f627e6ae585774b8fd2"
server_key_version: "encryptionkey@0"
I20260927 13:54:27.487126 7863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:54:27.490903 7878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:27.492478 7863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:54:27.492683 7863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "548510bc8cf144e1970d2f13d02316cf"
format_stamp: "Formatted at 2026-09-27 13:54:27 on dist-test-slave-q25r"
server_key: "6aa15e84a73c64188b3a99caadb5136f"
server_key_iv: "18d1b589eb866f627e6ae585774b8fd2"
server_key_version: "encryptionkey@0"
I20260927 13:54:27.492899 7863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:27.518630 7863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:27.523073 7863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:27.523397 7863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:27.541920 7863 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:44805
I20260927 13:54:27.541965 7929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:44805 every 8 connection(s)
I20260927 13:54:27.543326 7863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:27.547268 7930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:27.547623 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 7863
I20260927 13:54:27.547821 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:27.548434 21013 external_mini_cluster.cc:1428] Setting key 408b74ae8d164e32a110b3e0879f3945
I20260927 13:54:27.561939 7930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf: Bootstrap starting.
Sep 27 13:54:27 dist-test-slave-q25r krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:27.566017 7930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:27.567130 7930 log.cc:824] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:27.570561 7930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf: No bootstrap required, opened a new log
I20260927 13:54:27.574517 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:27.550451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.20.133.126:44805)
0927 13:54:27.551260 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:27.551271 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:27.551312 (+ 41us) server_negotiation.cc:407] Connection header received
0927 13:54:27.553680 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:27.553703 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:27.554103 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:27.554433 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:27.556493 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:27.557811 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:27.559910 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:27.560721 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:27.560859 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:27.565918 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:27.565927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:27.565945 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:27.565988 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:27.569571 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:27.570381 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:27.570391 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:27.570399 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:27.570623 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:27.571104 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:27.571109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:27.571113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:27.571851 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:27.572163 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:27.572650 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 13:54:27.573130 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":381,"threads_started":1}
I20260927 13:54:27.579521 7930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548510bc8cf144e1970d2f13d02316cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44805 } }
I20260927 13:54:27.579908 7930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:27.579988 7930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548510bc8cf144e1970d2f13d02316cf, State: Initialized, Role: FOLLOWER
I20260927 13:54:27.580535 7930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548510bc8cf144e1970d2f13d02316cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44805 } }
I20260927 13:54:27.580781 7930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:27.580896 7930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:27.581025 7930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:27.583559 7930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548510bc8cf144e1970d2f13d02316cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44805 } }
I20260927 13:54:27.584050 7930 leader_election.cc:302] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 548510bc8cf144e1970d2f13d02316cf; no voters:
I20260927 13:54:27.584679 7930 leader_election.cc:288] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:27.584820 7935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:27.585448 7935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [term 1 LEADER]: Becoming Leader. State: Replica: 548510bc8cf144e1970d2f13d02316cf, State: Running, Role: LEADER
I20260927 13:54:27.585999 7935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548510bc8cf144e1970d2f13d02316cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44805 } }
I20260927 13:54:27.586740 7930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:27.589221 7936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "548510bc8cf144e1970d2f13d02316cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548510bc8cf144e1970d2f13d02316cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44805 } } }
I20260927 13:54:27.589389 7937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 548510bc8cf144e1970d2f13d02316cf. Latest consensus state: current_term: 1 leader_uuid: "548510bc8cf144e1970d2f13d02316cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548510bc8cf144e1970d2f13d02316cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44805 } } }
I20260927 13:54:27.589659 7937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:27.589659 7936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:27.590618 7943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:27.605497 7943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:27.616156 7943 catalog_manager.cc:1347] Generated new cluster ID: 7ae3221f838948279e6a6f08e797b3e7
I20260927 13:54:27.616379 7943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:27.628194 7943 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:27.630349 7943 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:27.643908 7943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf: Generated new TSK 0
I20260927 13:54:27.645166 7943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.133.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
Entry for principal kudu/127.20.133.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab.
I20260927 13:54:27.704820 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:44805
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:28.043910 7956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:28.044214 7956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:28.044349 7956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:28.051807 7956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:28.051939 7956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:28.052083 7956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:54:28.062119 7956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.65.keytab
--principal=kudu/127.20.133.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:28.063570 7956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:28.064985 7956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:28.078356 7961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:28.078320 7964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:28.078379 7962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:28.079217 7956 server_base.cc:1034] running on GCE node
I20260927 13:54:28.079917 7956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:28.080976 7956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:28.082231 7956 hybrid_clock.cc:648] HybridClock initialized: now 1790517268082163 us; error 54 us; skew 500 ppm
I20260927 13:54:28.082574 7956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:28 dist-test-slave-q25r krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517268, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:28.085855 7956 init.cc:377] Logged in from keytab as kudu/127.20.133.65@KRBTEST.COM (short username kudu)
I20260927 13:54:28.087443 7956 webserver.cc:460] Webserver started at http://127.20.133.65:40335/ using document root <none> and password file <none>
I20260927 13:54:28.088140 7956 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:28.088259 7956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:28.088670 7956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:28.091619 7956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "9505817b69534dd994fcbceb1f6a7c6f"
format_stamp: "Formatted at 2026-09-27 13:54:28 on dist-test-slave-q25r"
server_key: "a3ed49ddbce500550f22b43a811c1b86"
server_key_iv: "a4f7b10910c6273f5742fcde87ea13f4"
server_key_version: "encryptionkey@0"
I20260927 13:54:28.092471 7956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "9505817b69534dd994fcbceb1f6a7c6f"
format_stamp: "Formatted at 2026-09-27 13:54:28 on dist-test-slave-q25r"
server_key: "a3ed49ddbce500550f22b43a811c1b86"
server_key_iv: "a4f7b10910c6273f5742fcde87ea13f4"
server_key_version: "encryptionkey@0"
I20260927 13:54:28.098073 7956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 13:54:28.101995 7971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:28.103610 7956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:54:28.103806 7956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "9505817b69534dd994fcbceb1f6a7c6f"
format_stamp: "Formatted at 2026-09-27 13:54:28 on dist-test-slave-q25r"
server_key: "a3ed49ddbce500550f22b43a811c1b86"
server_key_iv: "a4f7b10910c6273f5742fcde87ea13f4"
server_key_version: "encryptionkey@0"
I20260927 13:54:28.104034 7956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:28.141268 7956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:28.145627 7956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:28.145901 7956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:28.146931 7956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:28.148294 7956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:28.148411 7956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:28.148537 7956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:28.148592 7956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:28.181566 7956 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:40761
I20260927 13:54:28.181629 8083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:40761 every 8 connection(s)
I20260927 13:54:28.182921 7956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:54:28.190920 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 7956
I20260927 13:54:28.191102 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:54:28.191568 21013 external_mini_cluster.cc:1428] Setting key 89c763f796cf2a7f25089e10ab3631ac
Sep 27 13:54:28 dist-test-slave-q25r krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517268, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.65@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:28.205724 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:28.185813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:47275 (local address 127.20.133.126:44805)
0927 13:54:28.186341 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:28.186346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:28.187336 (+ 990us) server_negotiation.cc:407] Connection header received
0927 13:54:28.189487 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:28.189493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:28.189584 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:28.189680 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:28.191496 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:28.192297 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:28.194232 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:28.194672 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:28.194790 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:28.198678 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:28.198685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:28.198690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:28.198736 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:28.202056 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:28.203214 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:28.203218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:28.203220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:28.203343 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:28.203876 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:28.203880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:28.203882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:28.204149 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:28.204272 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:28.205255 (+ 983us) server_negotiation.cc:299] Negotiation successful
0927 13:54:28.205447 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":265,"threads_started":1}
I20260927 13:54:28.207176 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:28.186366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:44805 (local address 127.20.133.65:47275)
0927 13:54:28.187160 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:28.187202 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:28.189162 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:28.189955 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:28.189966 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:28.190564 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:28.191290 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:28.191319 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:28.192503 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:28.192507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:28.194067 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:28.194082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:28.194934 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:28.194941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:28.195186 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:28.195858 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:54:28.195905 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:28.198370 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:28.202278 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:28.202291 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:28.202313 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:28.203052 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:28.203491 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:28.203497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:28.203500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:28.203769 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:28.204287 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:28.204296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:28.204825 (+ 529us) client_negotiation.cc:769] Sending connection context
0927 13:54:28.205161 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 13:54:28.205765 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":309,"threads_started":1}
I20260927 13:54:28.208889 8084 heartbeater.cc:347] Connected to a master server at 127.20.133.126:44805
I20260927 13:54:28.209227 8084 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:28.209959 8084 heartbeater.cc:511] Master 127.20.133.126:44805 requested a full tablet report, sending...
I20260927 13:54:28.212193 7895 ts_manager.cc:194] Registered new tserver with Master: 9505817b69534dd994fcbceb1f6a7c6f (127.20.133.65:40761)
I20260927 13:54:28.213766 7895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.65@KRBTEST.COM'} at 127.20.133.65:47275
WARNING: no policy specified for kudu/127.20.133.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
Entry for principal kudu/127.20.133.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab.
I20260927 13:54:28.251343 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:44805
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:28.584425 8090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:28.584733 8090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:28.584830 8090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:28.592381 8090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:28.592512 8090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:28.592655 8090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:54:28.603273 8090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.66.keytab
--principal=kudu/127.20.133.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:28.604645 8090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:28.605929 8090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:28.619407 8095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:28.619446 8096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:28.619468 8098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:28.620178 8090 server_base.cc:1034] running on GCE node
I20260927 13:54:28.620913 8090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:28.621963 8090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:28.623220 8090 hybrid_clock.cc:648] HybridClock initialized: now 1790517268623168 us; error 39 us; skew 500 ppm
I20260927 13:54:28.623621 8090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:28 dist-test-slave-q25r krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517268, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:28.627023 8090 init.cc:377] Logged in from keytab as kudu/127.20.133.66@KRBTEST.COM (short username kudu)
I20260927 13:54:28.628639 8090 webserver.cc:460] Webserver started at http://127.20.133.66:39133/ using document root <none> and password file <none>
I20260927 13:54:28.629350 8090 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:28.629477 8090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:28.629807 8090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:28.633152 8090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "40bb606479c142ffa35ca68ec21c6bc7"
format_stamp: "Formatted at 2026-09-27 13:54:28 on dist-test-slave-q25r"
server_key: "4dc9415d4f72f1b0599f6eea61847386"
server_key_iv: "77de6e8111f11a9b530405aa729b0297"
server_key_version: "encryptionkey@0"
I20260927 13:54:28.634017 8090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "40bb606479c142ffa35ca68ec21c6bc7"
format_stamp: "Formatted at 2026-09-27 13:54:28 on dist-test-slave-q25r"
server_key: "4dc9415d4f72f1b0599f6eea61847386"
server_key_iv: "77de6e8111f11a9b530405aa729b0297"
server_key_version: "encryptionkey@0"
I20260927 13:54:28.639758 8090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 13:54:28.643517 8105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:28.645004 8090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:54:28.645184 8090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "40bb606479c142ffa35ca68ec21c6bc7"
format_stamp: "Formatted at 2026-09-27 13:54:28 on dist-test-slave-q25r"
server_key: "4dc9415d4f72f1b0599f6eea61847386"
server_key_iv: "77de6e8111f11a9b530405aa729b0297"
server_key_version: "encryptionkey@0"
I20260927 13:54:28.645390 8090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:28.656704 8090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:28.661397 8090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:28.661695 8090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:28.662726 8090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:28.664147 8090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:28.664230 8090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:28.664371 8090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:28.664433 8090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 13:54:28.695739 8090 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:35801
I20260927 13:54:28.695775 8217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:35801 every 8 connection(s)
I20260927 13:54:28.697146 8090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:54:28.706643 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 8090
I20260927 13:54:28.706799 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:54:28.707209 21013 external_mini_cluster.cc:1428] Setting key 67e36b776558db9a73b544c04bae59ac
Sep 27 13:54:28 dist-test-slave-q25r krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517268, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.66@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:28.717052 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:28.699731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:55925 (local address 127.20.133.126:44805)
0927 13:54:28.699910 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:28.699914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:28.701069 (+ 1155us) server_negotiation.cc:407] Connection header received
0927 13:54:28.702919 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:28.702925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:28.703021 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:28.703106 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:28.704759 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:28.705478 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:28.707126 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:28.707638 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:28.707757 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:28.711457 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:28.711462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:28.711466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:28.711499 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:28.713926 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:28.714730 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:28.714734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:28.714737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:28.714869 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:28.715298 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:28.715302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:28.715304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:28.715575 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:28.715711 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:28.716469 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 13:54:28.716719 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:54:28.718047 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:28.700160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:44805 (local address 127.20.133.66:55925)
0927 13:54:28.700897 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:28.700937 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:28.702627 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:28.703378 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:28.703386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:28.703920 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:28.704564 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:28.704585 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:28.705643 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:28.705647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:28.706969 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:28.706982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:28.707871 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:28.707880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:28.708214 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:28.708987 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:54:28.709010 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:28.711217 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:28.714074 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:28.714084 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:28.714105 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:28.714590 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:28.714981 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:28.714986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:28.714990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:28.715177 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:28.715712 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:28.715721 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:28.716069 (+ 348us) client_negotiation.cc:769] Sending connection context
0927 13:54:28.716312 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 13:54:28.716905 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":299,"threads_started":1}
I20260927 13:54:28.719475 8218 heartbeater.cc:347] Connected to a master server at 127.20.133.126:44805
I20260927 13:54:28.719812 8218 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:28.720661 8218 heartbeater.cc:511] Master 127.20.133.126:44805 requested a full tablet report, sending...
I20260927 13:54:28.722255 7895 ts_manager.cc:194] Registered new tserver with Master: 40bb606479c142ffa35ca68ec21c6bc7 (127.20.133.66:35801)
I20260927 13:54:28.723312 7895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.66@KRBTEST.COM'} at 127.20.133.66:55925
WARNING: no policy specified for kudu/127.20.133.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
Entry for principal kudu/127.20.133.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab.
I20260927 13:54:28.763948 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:44805
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:29.110117 8223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:29.110425 8223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:29.110523 8223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:29.118186 8223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:29.118327 8223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:29.118445 8223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:54:29.129215 8223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.67.keytab
--principal=kudu/127.20.133.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:44805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:29.130504 8223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:29.131806 8223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:29.146195 8229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:29.146236 8228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:29.146160 8231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:29.150480 8223 server_base.cc:1034] running on GCE node
I20260927 13:54:29.151185 8223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:29.152189 8223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:29.153440 8223 hybrid_clock.cc:648] HybridClock initialized: now 1790517269153381 us; error 46 us; skew 500 ppm
I20260927 13:54:29.153811 8223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:29 dist-test-slave-q25r krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517269, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:29.157101 8223 init.cc:377] Logged in from keytab as kudu/127.20.133.67@KRBTEST.COM (short username kudu)
I20260927 13:54:29.158814 8223 webserver.cc:460] Webserver started at http://127.20.133.67:39969/ using document root <none> and password file <none>
I20260927 13:54:29.159555 8223 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:29.159670 8223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:29.159977 8223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:29.163220 8223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "c8f95f950b7b491092eaf59fca282689"
format_stamp: "Formatted at 2026-09-27 13:54:29 on dist-test-slave-q25r"
server_key: "19400543224386d8d4b609d484623e59"
server_key_iv: "c593fbe36a8bea712c3c534795ef7a37"
server_key_version: "encryptionkey@0"
I20260927 13:54:29.164134 8223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "c8f95f950b7b491092eaf59fca282689"
format_stamp: "Formatted at 2026-09-27 13:54:29 on dist-test-slave-q25r"
server_key: "19400543224386d8d4b609d484623e59"
server_key_iv: "c593fbe36a8bea712c3c534795ef7a37"
server_key_version: "encryptionkey@0"
I20260927 13:54:29.169961 8223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:54:29.174156 8238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:29.176152 8223 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:54:29.176427 8223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "c8f95f950b7b491092eaf59fca282689"
format_stamp: "Formatted at 2026-09-27 13:54:29 on dist-test-slave-q25r"
server_key: "19400543224386d8d4b609d484623e59"
server_key_iv: "c593fbe36a8bea712c3c534795ef7a37"
server_key_version: "encryptionkey@0"
I20260927 13:54:29.176728 8223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:29.192404 8223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:29.196787 8223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:29.197052 8223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:29.198100 8223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:29.199529 8223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:29.199615 8223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:29.199713 8223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:29.199775 8223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:29.217113 8084 heartbeater.cc:503] Master 127.20.133.126:44805 was elected leader, sending a full tablet report...
I20260927 13:54:29.232604 8223 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:40377
I20260927 13:54:29.232640 8350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:40377 every 8 connection(s)
I20260927 13:54:29.233940 8223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:54:29.240715 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 8223
I20260927 13:54:29.240890 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:54:29.241338 21013 external_mini_cluster.cc:1428] Setting key 336a2f690869acf2fe9c23feae481473
Sep 27 13:54:29 dist-test-slave-q25r krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517269, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.67@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:29.254487 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.236540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:56605 (local address 127.20.133.126:44805)
0927 13:54:29.237000 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:29.237004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:29.237978 (+ 974us) server_negotiation.cc:407] Connection header received
0927 13:54:29.239901 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:29.239907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:29.239988 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:29.240085 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:29.241829 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.242792 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.244482 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.244896 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.245010 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.248679 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:29.248684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:29.248687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:29.248725 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:29.251221 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:29.251997 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:29.252003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:29.252006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:29.252136 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:29.252661 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:29.252665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:29.252667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:29.252954 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:29.253096 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:29.253857 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 13:54:29.254109 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":222,"threads_started":1}
I20260927 13:54:29.255609 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.237058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:44805 (local address 127.20.133.67:56605)
0927 13:54:29.237815 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:29.237854 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:29.239605 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:29.240400 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:29.240409 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:29.240983 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:29.241627 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.241649 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.242964 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.242968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.244295 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.244309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.245144 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.245151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.245394 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.246070 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:54:29.246096 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:29.248387 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:29.251390 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:29.251402 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:29.251417 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:29.251853 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:29.252250 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:54:29.252255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:54:29.252259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:54:29.252525 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:54:29.253091 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:29.253099 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:29.253470 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 13:54:29.253722 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 13:54:29.254311 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":255,"threads_started":1}
I20260927 13:54:29.257099 8351 heartbeater.cc:347] Connected to a master server at 127.20.133.126:44805
I20260927 13:54:29.257391 8351 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:29.258191 8351 heartbeater.cc:511] Master 127.20.133.126:44805 requested a full tablet report, sending...
I20260927 13:54:29.259828 7894 ts_manager.cc:194] Registered new tserver with Master: c8f95f950b7b491092eaf59fca282689 (127.20.133.67:40377)
I20260927 13:54:29.260824 7894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.133.67@KRBTEST.COM'} at 127.20.133.67:56605
I20260927 13:54:29.269845 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:54:29.287505 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.274323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.20.133.126:44805)
0927 13:54:29.274586 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:29.274592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:29.274822 (+ 230us) server_negotiation.cc:407] Connection header received
0927 13:54:29.274971 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:29.274977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:29.275067 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:29.275156 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:29.276698 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.277463 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.278905 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.279380 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.279484 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.281999 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:29.282005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:29.282008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:29.282049 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:29.284717 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:29.285520 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:29.285525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:29.285527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:29.285669 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:29.286004 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:29.286007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:29.286009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:29.286353 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:29.286559 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:29.287059 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 13:54:29.287252 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 13:54:29.291890 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:54:29.299540 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:54:29.336359 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.313034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.20.133.67:40377)
0927 13:54:29.327351 (+ 14317us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:29.327360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:29.327400 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:54:29.327511 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:29.327533 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:29.327820 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:29.328043 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:29.328925 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.330499 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.333529 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.335068 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.335207 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.335461 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:29.335798 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 13:54:29.335981 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14137,"thread_start_us":13997,"threads_started":1}
I20260927 13:54:29.336395 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.312687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:40377 (local address 127.0.0.1:35628)
0927 13:54:29.326068 (+ 13381us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:29.326122 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:29.326680 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:29.328137 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:29.328147 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:29.328189 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:29.328708 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.328721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.330663 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.330668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.333319 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.333333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.335297 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.335304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.335498 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.335566 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 13:54:29.335703 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 13:54:29.335906 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13078,"mutex_wait_us":21,"thread_start_us":16387,"threads_started":1}
I20260927 13:54:29.337446 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.315109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.20.133.66:35801)
0927 13:54:29.328776 (+ 13667us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:29.328784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:29.329631 (+ 847us) server_negotiation.cc:407] Connection header received
0927 13:54:29.329732 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:29.329739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:29.329977 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:29.330172 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:29.331093 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.332111 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.334570 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.335715 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.335843 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.336009 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:29.336939 (+ 930us) server_negotiation.cc:299] Negotiation successful
0927 13:54:29.337127 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13517,"thread_start_us":13379,"threads_started":1}
I20260927 13:54:29.338229 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.314796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:35801 (local address 127.0.0.1:52072)
0927 13:54:29.329371 (+ 14575us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:29.329388 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:29.329507 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:29.330373 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:29.330377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:29.330401 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:29.330856 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.330869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.332254 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.332260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.334403 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.334415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.336157 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.336163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.336390 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.336422 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:54:29.336507 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 13:54:29.337634 (+ 1127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14435,"mutex_wait_us":58,"thread_start_us":14029,"threads_started":1}
I20260927 13:54:29.338614 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.313022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.20.133.65:40761)
0927 13:54:29.325398 (+ 12376us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:29.325410 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:29.329327 (+ 3917us) server_negotiation.cc:407] Connection header received
0927 13:54:29.329649 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:29.329657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:29.329921 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:29.330099 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:29.331142 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.332797 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.335377 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.336769 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.336880 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.337075 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:29.338131 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0927 13:54:29.338310 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12128,"thread_start_us":13328,"threads_started":1}
I20260927 13:54:29.339089 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.312808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40761 (local address 127.0.0.1:54088)
0927 13:54:29.329190 (+ 16382us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:29.329211 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:29.329449 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:29.330361 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:29.330365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:29.330391 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:29.330976 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.330991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.332964 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.332970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.335198 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.335212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.337763 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.337769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.337943 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.337974 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 13:54:29.338074 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 13:54:29.338206 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16233,"mutex_wait_us":1,"thread_start_us":12821,"threads_started":1}
I20260927 13:54:29.340698 8286 tablet_service.cc:1467] Processing CreateTablet for tablet 9621ba7e503a407798eed16eac89ef70 (DEFAULT_TABLE table=test-table [id=67e7ab4161ef4b38ac30bb09cef9226d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:29.342130 8286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9621ba7e503a407798eed16eac89ef70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:29.341745 8153 tablet_service.cc:1467] Processing CreateTablet for tablet 9621ba7e503a407798eed16eac89ef70 (DEFAULT_TABLE table=test-table [id=67e7ab4161ef4b38ac30bb09cef9226d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:29.342414 8019 tablet_service.cc:1467] Processing CreateTablet for tablet 9621ba7e503a407798eed16eac89ef70 (DEFAULT_TABLE table=test-table [id=67e7ab4161ef4b38ac30bb09cef9226d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:29.343106 8153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9621ba7e503a407798eed16eac89ef70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:29.343717 8019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9621ba7e503a407798eed16eac89ef70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:29.355424 8370 tablet_bootstrap.cc:492] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: Bootstrap starting.
I20260927 13:54:29.357645 8371 tablet_bootstrap.cc:492] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: Bootstrap starting.
I20260927 13:54:29.359541 8372 tablet_bootstrap.cc:492] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: Bootstrap starting.
I20260927 13:54:29.361316 8370 tablet_bootstrap.cc:654] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:29.361320 8371 tablet_bootstrap.cc:654] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:29.362447 8372 tablet_bootstrap.cc:654] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:29.362684 8371 log.cc:824] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:29.362684 8370 log.cc:824] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:29.363535 8372 log.cc:824] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:29.366127 8370 tablet_bootstrap.cc:492] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: No bootstrap required, opened a new log
I20260927 13:54:29.366178 8371 tablet_bootstrap.cc:492] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: No bootstrap required, opened a new log
I20260927 13:54:29.366410 8370 ts_tablet_manager.cc:1397] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 13:54:29.366451 8371 ts_tablet_manager.cc:1397] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 13:54:29.367264 8372 tablet_bootstrap.cc:492] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: No bootstrap required, opened a new log
I20260927 13:54:29.367545 8372 ts_tablet_manager.cc:1397] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260927 13:54:29.375211 8370 raft_consensus.cc:348] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f95f950b7b491092eaf59fca282689" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 40377 } } peers { permanent_uuid: "9505817b69534dd994fcbceb1f6a7c6f" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40761 } } peers { permanent_uuid: "40bb606479c142ffa35ca68ec21c6bc7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35801 } }
I20260927 13:54:29.375650 8370 raft_consensus.cc:374] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:29.375746 8370 raft_consensus.cc:729] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bb606479c142ffa35ca68ec21c6bc7, State: Initialized, Role: FOLLOWER
I20260927 13:54:29.376448 8370 consensus_queue.cc:260] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f95f950b7b491092eaf59fca282689" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 40377 } } peers { permanent_uuid: "9505817b69534dd994fcbceb1f6a7c6f" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40761 } } peers { permanent_uuid: "40bb606479c142ffa35ca68ec21c6bc7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35801 } }
I20260927 13:54:29.377208 8371 raft_consensus.cc:348] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f95f950b7b491092eaf59fca282689" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 40377 } } peers { permanent_uuid: "9505817b69534dd994fcbceb1f6a7c6f" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40761 } } peers { permanent_uuid: "40bb606479c142ffa35ca68ec21c6bc7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35801 } }
I20260927 13:54:29.377645 8218 heartbeater.cc:503] Master 127.20.133.126:44805 was elected leader, sending a full tablet report...
I20260927 13:54:29.377703 8371 raft_consensus.cc:374] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:29.377810 8371 raft_consensus.cc:729] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f95f950b7b491092eaf59fca282689, State: Initialized, Role: FOLLOWER
I20260927 13:54:29.378479 8371 consensus_queue.cc:260] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f95f950b7b491092eaf59fca282689" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 40377 } } peers { permanent_uuid: "9505817b69534dd994fcbceb1f6a7c6f" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40761 } } peers { permanent_uuid: "40bb606479c142ffa35ca68ec21c6bc7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35801 } }
I20260927 13:54:29.378489 8372 raft_consensus.cc:348] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f95f950b7b491092eaf59fca282689" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 40377 } } peers { permanent_uuid: "9505817b69534dd994fcbceb1f6a7c6f" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40761 } } peers { permanent_uuid: "40bb606479c142ffa35ca68ec21c6bc7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35801 } }
I20260927 13:54:29.378925 8372 raft_consensus.cc:374] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:29.379031 8372 raft_consensus.cc:729] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9505817b69534dd994fcbceb1f6a7c6f, State: Initialized, Role: FOLLOWER
I20260927 13:54:29.380010 8370 ts_tablet_manager.cc:1428] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 13:54:29.379843 8372 consensus_queue.cc:260] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f95f950b7b491092eaf59fca282689" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 40377 } } peers { permanent_uuid: "9505817b69534dd994fcbceb1f6a7c6f" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40761 } } peers { permanent_uuid: "40bb606479c142ffa35ca68ec21c6bc7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35801 } }
I20260927 13:54:29.381078 8351 heartbeater.cc:503] Master 127.20.133.126:44805 was elected leader, sending a full tablet report...
I20260927 13:54:29.381793 8371 ts_tablet_manager.cc:1428] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260927 13:54:29.381942 8372 ts_tablet_manager.cc:1428] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
W20260927 13:54:29.435513 8085 tablet.cc:2369] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:54:29.449231 8219 tablet.cc:2369] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:54:29.485622 8352 tablet.cc:2369] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:54:29.519137 8376 raft_consensus.cc:482] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:54:29.519435 8376 raft_consensus.cc:504] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f95f950b7b491092eaf59fca282689" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 40377 } } peers { permanent_uuid: "9505817b69534dd994fcbceb1f6a7c6f" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40761 } } peers { permanent_uuid: "40bb606479c142ffa35ca68ec21c6bc7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35801 } }
I20260927 13:54:29.521188 8376 leader_election.cc:288] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f95f950b7b491092eaf59fca282689 (127.20.133.67:40377), 9505817b69534dd994fcbceb1f6a7c6f (127.20.133.65:40761)
I20260927 13:54:29.530650 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.521717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:55307 (local address 127.20.133.67:40377)
0927 13:54:29.521922 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:29.521928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:29.522800 (+ 872us) server_negotiation.cc:407] Connection header received
0927 13:54:29.522992 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:29.522997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:29.523061 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:29.523136 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:29.523920 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.525152 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.528169 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.529452 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.529609 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.529689 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:29.530236 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 13:54:29.530388 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:54:29.530673 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.521845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:60611 (local address 127.20.133.65:40761)
0927 13:54:29.522081 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:29.522089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:29.522866 (+ 777us) server_negotiation.cc:407] Connection header received
0927 13:54:29.523046 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:29.523051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:29.523131 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:29.523250 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:29.524096 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.525154 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.528179 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.529432 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.529628 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.529693 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:29.530236 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 13:54:29.530392 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 13:54:29.530820 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.521694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:40761 (local address 127.20.133.66:60611)
0927 13:54:29.522737 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:29.522754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:29.522899 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:29.523342 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:29.523349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:29.523380 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:29.523827 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.523838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.525318 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.525324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.527946 (+ 2622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.527968 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.529741 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.529747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.529992 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.530043 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 13:54:29.530148 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 13:54:29.530321 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":198,"thread_start_us":661,"threads_started":1}
I20260927 13:54:29.530844 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.521685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:40377 (local address 127.20.133.66:55307)
0927 13:54:29.522669 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:29.522690 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:29.522853 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:29.523185 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:29.523189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:29.523223 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:29.523701 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.523718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.525317 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.525324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.527946 (+ 2622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:29.527968 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.529732 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.529740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:29.529986 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.530039 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 13:54:29.530148 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 13:54:29.530321 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":160,"spinlock_wait_cycles":24960,"thread_start_us":416,"threads_started":1}
I20260927 13:54:29.532186 8039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9621ba7e503a407798eed16eac89ef70" candidate_uuid: "40bb606479c142ffa35ca68ec21c6bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9505817b69534dd994fcbceb1f6a7c6f" is_pre_election: true
I20260927 13:54:29.532158 8306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9621ba7e503a407798eed16eac89ef70" candidate_uuid: "40bb606479c142ffa35ca68ec21c6bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f95f950b7b491092eaf59fca282689" is_pre_election: true
I20260927 13:54:29.532806 8039 raft_consensus.cc:2454] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40bb606479c142ffa35ca68ec21c6bc7 in term 0.
I20260927 13:54:29.532832 8306 raft_consensus.cc:2454] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40bb606479c142ffa35ca68ec21c6bc7 in term 0.
I20260927 13:54:29.533550 8108 leader_election.cc:302] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bb606479c142ffa35ca68ec21c6bc7, 9505817b69534dd994fcbceb1f6a7c6f; no voters:
I20260927 13:54:29.534004 8376 raft_consensus.cc:2781] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:54:29.534168 8376 raft_consensus.cc:482] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:54:29.534267 8376 raft_consensus.cc:3037] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:29.536453 8376 raft_consensus.cc:504] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f95f950b7b491092eaf59fca282689" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 40377 } } peers { permanent_uuid: "9505817b69534dd994fcbceb1f6a7c6f" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40761 } } peers { permanent_uuid: "40bb606479c142ffa35ca68ec21c6bc7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35801 } }
I20260927 13:54:29.537091 8376 leader_election.cc:288] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [CANDIDATE]: Term 1 election: Requested vote from peers c8f95f950b7b491092eaf59fca282689 (127.20.133.67:40377), 9505817b69534dd994fcbceb1f6a7c6f (127.20.133.65:40761)
I20260927 13:54:29.537739 8039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9621ba7e503a407798eed16eac89ef70" candidate_uuid: "40bb606479c142ffa35ca68ec21c6bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9505817b69534dd994fcbceb1f6a7c6f"
I20260927 13:54:29.537732 8306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9621ba7e503a407798eed16eac89ef70" candidate_uuid: "40bb606479c142ffa35ca68ec21c6bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f95f950b7b491092eaf59fca282689"
I20260927 13:54:29.537977 8306 raft_consensus.cc:3037] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:29.537978 8039 raft_consensus.cc:3037] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:29.541360 8039 raft_consensus.cc:2454] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40bb606479c142ffa35ca68ec21c6bc7 in term 1.
I20260927 13:54:29.541360 8306 raft_consensus.cc:2454] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40bb606479c142ffa35ca68ec21c6bc7 in term 1.
I20260927 13:54:29.542734 8108 leader_election.cc:302] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bb606479c142ffa35ca68ec21c6bc7, 9505817b69534dd994fcbceb1f6a7c6f; no voters:
I20260927 13:54:29.543432 8376 raft_consensus.cc:2781] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:29.544287 8376 raft_consensus.cc:686] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 1 LEADER]: Becoming Leader. State: Replica: 40bb606479c142ffa35ca68ec21c6bc7, State: Running, Role: LEADER
I20260927 13:54:29.545004 8376 consensus_queue.cc:237] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f95f950b7b491092eaf59fca282689" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 40377 } } peers { permanent_uuid: "9505817b69534dd994fcbceb1f6a7c6f" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40761 } } peers { permanent_uuid: "40bb606479c142ffa35ca68ec21c6bc7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35801 } }
I20260927 13:54:29.549902 7894 catalog_manager.cc:5495] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40bb606479c142ffa35ca68ec21c6bc7 (127.20.133.66). New cstate: current_term: 1 leader_uuid: "40bb606479c142ffa35ca68ec21c6bc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f95f950b7b491092eaf59fca282689" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9505817b69534dd994fcbceb1f6a7c6f" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40bb606479c142ffa35ca68ec21c6bc7" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35801 } health_report { overall_health: HEALTHY } } }
I20260927 13:54:29.586097 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.580296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.20.133.66:35801)
0927 13:54:29.580544 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:29.580548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:29.580568 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:54:29.580825 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:29.580830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:29.580915 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:29.581051 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:54:29.581478 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.582286 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.583723 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.584154 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.584253 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.584927 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:54:29.585462 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:54:29.585597 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:29.585732 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 13:54:29.585809 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 13:54:29.594883 8306 raft_consensus.cc:1261] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689 [term 1 FOLLOWER]: Refusing update from remote peer 40bb606479c142ffa35ca68ec21c6bc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:29.594883 8039 raft_consensus.cc:1261] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f [term 1 FOLLOWER]: Refusing update from remote peer 40bb606479c142ffa35ca68ec21c6bc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:29.596064 8376 consensus_queue.cc:1035] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f95f950b7b491092eaf59fca282689" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:29.596424 8381 consensus_queue.cc:1035] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9505817b69534dd994fcbceb1f6a7c6f" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 40761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:29.607103 8384 mvcc.cc:205] Tried to move back new op lower bound from 7333958736249118720 to 7333958736060522496. Current Snapshot: MvccSnapshot[applied={T|T < 7333958736249118720}]
I20260927 13:54:29.608172 8386 mvcc.cc:205] Tried to move back new op lower bound from 7333958736249118720 to 7333958736060522496. Current Snapshot: MvccSnapshot[applied={T|T < 7333958736249118720}]
I20260927 13:54:29.623741 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:29.624079 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:29.628621 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 548510bc8cf144e1970d2f13d02316cf: Sending DeleteTablet for 3 replicas of tablet 9621ba7e503a407798eed16eac89ef70
I20260927 13:54:29.629977 8286 tablet_service.cc:1514] Processing DeleteTablet for tablet 9621ba7e503a407798eed16eac89ef70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:29 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:35628
I20260927 13:54:29.630126 8153 tablet_service.cc:1514] Processing DeleteTablet for tablet 9621ba7e503a407798eed16eac89ef70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:29 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:52072
I20260927 13:54:29.630162 8019 tablet_service.cc:1514] Processing DeleteTablet for tablet 9621ba7e503a407798eed16eac89ef70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:29 UTC) from {username='kudu', principal='kudu/127.20.133.126@KRBTEST.COM'} at 127.0.0.1:54088
I20260927 13:54:29.632222 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 7956
I20260927 13:54:29.642488 8396 tablet_replica.cc:330] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: stopping tablet replica
I20260927 13:54:29.643052 8396 raft_consensus.cc:2229] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:29.643702 8396 raft_consensus.cc:2258] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:29.644646 8395 tablet_replica.cc:330] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: stopping tablet replica
I20260927 13:54:29.645162 8395 raft_consensus.cc:2229] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:54:29.645931 8395 raft_consensus.cc:2258] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:29.649840 8395 ts_tablet_manager.cc:1905] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:29.657137 8396 ts_tablet_manager.cc:1905] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:29.658334 8397 tablet_replica.cc:330] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: stopping tablet replica
I20260927 13:54:29.659276 8397 raft_consensus.cc:2229] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:29.659870 8397 raft_consensus.cc:2258] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 13:54:29 dist-test-slave-q25r krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.65@KRBTEST.COM
I20260927 13:54:29.662256 8397 ts_tablet_manager.cc:1905] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:29.665221 8396 ts_tablet_manager.cc:1918] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:29.665406 8396 log.cc:1196] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/9621ba7e503a407798eed16eac89ef70
I20260927 13:54:29.666213 8396 ts_tablet_manager.cc:1939] T 9621ba7e503a407798eed16eac89ef70 P c8f95f950b7b491092eaf59fca282689: Deleting consensus metadata
I20260927 13:54:29.668356 7882 catalog_manager.cc:4841] TS c8f95f950b7b491092eaf59fca282689 (127.20.133.67:40377): tablet 9621ba7e503a407798eed16eac89ef70 (table test-table [id=67e7ab4161ef4b38ac30bb09cef9226d]) successfully deleted
I20260927 13:54:29.669817 8397 ts_tablet_manager.cc:1918] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:29.670001 8397 log.cc:1196] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-0/wal/wals/9621ba7e503a407798eed16eac89ef70
I20260927 13:54:29.670552 8397 ts_tablet_manager.cc:1939] T 9621ba7e503a407798eed16eac89ef70 P 9505817b69534dd994fcbceb1f6a7c6f: Deleting consensus metadata
I20260927 13:54:29.670861 8395 ts_tablet_manager.cc:1918] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:29.671038 8395 log.cc:1196] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/9621ba7e503a407798eed16eac89ef70
I20260927 13:54:29.671590 8395 ts_tablet_manager.cc:1939] T 9621ba7e503a407798eed16eac89ef70 P 40bb606479c142ffa35ca68ec21c6bc7: Deleting consensus metadata
I20260927 13:54:29.672698 7881 catalog_manager.cc:4841] TS 9505817b69534dd994fcbceb1f6a7c6f (127.20.133.65:40761): tablet 9621ba7e503a407798eed16eac89ef70 (table test-table [id=67e7ab4161ef4b38ac30bb09cef9226d]) successfully deleted
I20260927 13:54:29.673046 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.634073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.20.133.65:40761)
0927 13:54:29.634300 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:29.634307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:29.649521 (+ 15214us) server_negotiation.cc:407] Connection header received
0927 13:54:29.650285 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:29.650291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:29.650397 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:29.650482 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:29.652599 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.653868 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.655955 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.656694 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.656809 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.663625 (+ 6816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:29.663647 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:29.663668 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:29.663734 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:29.668099 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:29.669437 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:29.669449 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:29.669452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:29.669660 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:29.670382 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:29.670388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:29.670391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:29.671071 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:29.671466 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:29.672215 (+ 749us) server_negotiation.cc:299] Negotiation successful
0927 13:54:29.672530 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 13:54:29.673362 7880 catalog_manager.cc:4841] TS 40bb606479c142ffa35ca68ec21c6bc7 (127.20.133.66:35801): tablet 9621ba7e503a407798eed16eac89ef70 (table test-table [id=67e7ab4161ef4b38ac30bb09cef9226d]) successfully deleted
I20260927 13:54:29.674818 8079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:29.794420 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 7956
I20260927 13:54:29.809002 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 8090
Sep 27 13:54:29 dist-test-slave-q25r krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.66@KRBTEST.COM
I20260927 13:54:29.825752 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.809865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.20.133.66:35801)
0927 13:54:29.810057 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:29.810063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:29.810081 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:54:29.810168 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:29.810171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:29.810249 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:29.810335 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:29.811590 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.812500 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.813771 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.814245 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.814356 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.819065 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:29.819082 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:29.819102 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:29.819165 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:29.822637 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:29.823483 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:29.823491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:29.823494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:29.823664 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:29.824049 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:29.824053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:29.824055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:29.824510 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:29.824877 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:29.825152 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 13:54:29.825392 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:54:29.826606 8213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:29.942153 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 8090
I20260927 13:54:29.959647 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 8223
Sep 27 13:54:29 dist-test-slave-q25r krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.67@KRBTEST.COM
I20260927 13:54:29.975361 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:29.960423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.20.133.67:40377)
0927 13:54:29.960600 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:29.960606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:29.960642 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:54:29.960820 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:29.960824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:29.960899 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:29.960986 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:29.962381 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.963100 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.964510 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:29.964916 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:29.965033 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:29.969634 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:29.969651 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:29.969667 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:29.969708 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:29.972405 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:29.973211 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:29.973219 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:29.973222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:29.973385 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:29.973794 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:29.973798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:29.973800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:29.974174 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:29.974468 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:29.974797 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 13:54:29.975029 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:54:29.976024 8346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:30.067198 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 8223
I20260927 13:54:30.080971 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 7863
I20260927 13:54:30.082360 7925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:30.196622 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 7863
2026-09-27T13:54:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4458 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T13:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:30Z Disabled control of system clock
I20260927 13:54:30.279732 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:34829
--webserver_interface=127.20.133.126
--webserver_port=0
--builtin_ntp_servers=127.20.133.84:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:34829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:54:30.683823 8406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:30.684257 8406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:30.684410 8406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:30.692698 8406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:30.692878 8406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:30.692939 8406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:30.693025 8406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:30.693089 8406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:30.703856 8406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:34829
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:34829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:30.705829 8406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:30.708308 8406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:30.722658 8411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:30.722764 8414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:30.722711 8412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:30.726408 8406 server_base.cc:1034] running on GCE node
I20260927 13:54:30.727445 8406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:30.729214 8406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:30.730446 8406 hybrid_clock.cc:648] HybridClock initialized: now 1790517270730388 us; error 63 us; skew 500 ppm
I20260927 13:54:30.730878 8406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:30.733774 8406 webserver.cc:460] Webserver started at http://127.20.133.126:38569/ using document root <none> and password file <none>
I20260927 13:54:30.734607 8406 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:30.734724 8406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:30.735059 8406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:30.738432 8406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "3f9ddb07643548ad85fb1411eb792bbd"
format_stamp: "Formatted at 2026-09-27 13:54:30 on dist-test-slave-q25r"
server_key: "cb04552b8d5257b9ac2cdbef0393d844"
server_key_iv: "bb915224cac3246b186aeb100878c6c8"
server_key_version: "encryptionkey@0"
I20260927 13:54:30.739339 8406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "3f9ddb07643548ad85fb1411eb792bbd"
format_stamp: "Formatted at 2026-09-27 13:54:30 on dist-test-slave-q25r"
server_key: "cb04552b8d5257b9ac2cdbef0393d844"
server_key_iv: "bb915224cac3246b186aeb100878c6c8"
server_key_version: "encryptionkey@0"
I20260927 13:54:30.745913 8406 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 13:54:30.750602 8420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:30.752651 8406 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 13:54:30.752902 8406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "3f9ddb07643548ad85fb1411eb792bbd"
format_stamp: "Formatted at 2026-09-27 13:54:30 on dist-test-slave-q25r"
server_key: "cb04552b8d5257b9ac2cdbef0393d844"
server_key_iv: "bb915224cac3246b186aeb100878c6c8"
server_key_version: "encryptionkey@0"
I20260927 13:54:30.753163 8406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:30.780781 8406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:30.781772 8406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:30.782078 8406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:30.802560 8406 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:34829
I20260927 13:54:30.802625 8471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:34829 every 8 connection(s)
I20260927 13:54:30.804049 8406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:30.808882 8472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:30.812538 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 8406
I20260927 13:54:30.812762 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:30.813396 21013 external_mini_cluster.cc:1428] Setting key e12e7f01a7787d938606f1c529b9f26e
I20260927 13:54:30.824455 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd: Bootstrap starting.
I20260927 13:54:30.829595 8472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:30.831118 8472 log.cc:824] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:30.832476 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:30.815948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.20.133.126:34829)
0927 13:54:30.817333 (+ 1385us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:30.817346 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:30.817398 (+ 52us) server_negotiation.cc:407] Connection header received
0927 13:54:30.819471 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:30.819509 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:30.820141 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:30.820665 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:30.821339 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:30.824228 (+ 2889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:30.826197 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:30.827117 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:30.827367 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:30.829203 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:30.829214 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:30.829236 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:30.830015 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:30.830171 (+ 156us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:30.830202 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:30.830362 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:30.830635 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 13:54:30.831046 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":535,"threads_started":1}
I20260927 13:54:30.834908 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd: No bootstrap required, opened a new log
I20260927 13:54:30.843834 8472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ddb07643548ad85fb1411eb792bbd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34829 } }
I20260927 13:54:30.844246 8472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:30.844326 8472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9ddb07643548ad85fb1411eb792bbd, State: Initialized, Role: FOLLOWER
I20260927 13:54:30.845156 8472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ddb07643548ad85fb1411eb792bbd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34829 } }
I20260927 13:54:30.845402 8472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:30.845520 8472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:30.845649 8472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:30.847975 8472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ddb07643548ad85fb1411eb792bbd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34829 } }
I20260927 13:54:30.848589 8472 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f9ddb07643548ad85fb1411eb792bbd; no voters:
I20260927 13:54:30.849227 8472 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:30.849504 8477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:30.850760 8477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [term 1 LEADER]: Becoming Leader. State: Replica: 3f9ddb07643548ad85fb1411eb792bbd, State: Running, Role: LEADER
I20260927 13:54:30.851558 8477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ddb07643548ad85fb1411eb792bbd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34829 } }
I20260927 13:54:30.852059 8472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:30.855530 8479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9ddb07643548ad85fb1411eb792bbd. Latest consensus state: current_term: 1 leader_uuid: "3f9ddb07643548ad85fb1411eb792bbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ddb07643548ad85fb1411eb792bbd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34829 } } }
I20260927 13:54:30.855535 8478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f9ddb07643548ad85fb1411eb792bbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ddb07643548ad85fb1411eb792bbd" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34829 } } }
I20260927 13:54:30.856125 8479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:30.856122 8478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:30.865389 8485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:30.871874 8485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:30.883123 8485 catalog_manager.cc:1347] Generated new cluster ID: 529548d261d246709ce56824d18c36fc
I20260927 13:54:30.883301 8485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:30.895702 8485 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:30.897146 8485 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:30.909644 8485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd: Generated new TSK 0
I20260927 13:54:30.910642 8485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:30.918540 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.65:0
--local_ip_for_outbound_sockets=127.20.133.65
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34829
--builtin_ntp_servers=127.20.133.84:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:54:31.297919 8496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:31.298236 8496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:31.298343 8496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:31.306082 8496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:31.306203 8496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:31.306348 8496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.65
I20260927 13:54:31.315877 8496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.133.65
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.65
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:31.317256 8496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:31.319199 8496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:31.333346 8501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:31.333310 8504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:31.333559 8502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:31.334282 8496 server_base.cc:1034] running on GCE node
I20260927 13:54:31.334959 8496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:31.336011 8496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:31.337251 8496 hybrid_clock.cc:648] HybridClock initialized: now 1790517271337197 us; error 50 us; skew 500 ppm
I20260927 13:54:31.337607 8496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:31.340583 8496 webserver.cc:460] Webserver started at http://127.20.133.65:46615/ using document root <none> and password file <none>
I20260927 13:54:31.341395 8496 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:31.341530 8496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:31.341892 8496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:31.345597 8496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/data/instance:
uuid: "7d0e59c8d53c4747bd20ac6fd4112d06"
format_stamp: "Formatted at 2026-09-27 13:54:31 on dist-test-slave-q25r"
server_key: "5b19bf07441e04fe7f92382bc14cf7cd"
server_key_iv: "9ff0ff0a58417400b1cdfb226ddb6ac3"
server_key_version: "encryptionkey@0"
I20260927 13:54:31.346539 8496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance:
uuid: "7d0e59c8d53c4747bd20ac6fd4112d06"
format_stamp: "Formatted at 2026-09-27 13:54:31 on dist-test-slave-q25r"
server_key: "5b19bf07441e04fe7f92382bc14cf7cd"
server_key_iv: "9ff0ff0a58417400b1cdfb226ddb6ac3"
server_key_version: "encryptionkey@0"
I20260927 13:54:31.352413 8496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 13:54:31.356578 8510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:31.358186 8496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:54:31.358347 8496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/wal
uuid: "7d0e59c8d53c4747bd20ac6fd4112d06"
format_stamp: "Formatted at 2026-09-27 13:54:31 on dist-test-slave-q25r"
server_key: "5b19bf07441e04fe7f92382bc14cf7cd"
server_key_iv: "9ff0ff0a58417400b1cdfb226ddb6ac3"
server_key_version: "encryptionkey@0"
I20260927 13:54:31.358560 8496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:31.373500 8496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:31.374327 8496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:31.374593 8496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:31.375553 8496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:31.377113 8496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:31.377204 8496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:31.377290 8496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:31.377341 8496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:31.406598 8496 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.65:33137
I20260927 13:54:31.406643 8622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.65:33137 every 8 connection(s)
I20260927 13:54:31.407904 8496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/data/info.pb
I20260927 13:54:31.414162 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 8496
I20260927 13:54:31.414351 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-0/wal/instance
I20260927 13:54:31.414923 21013 external_mini_cluster.cc:1428] Setting key 7133952d6e342ed455b81201eb66dde7
I20260927 13:54:31.417841 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.66:0
--local_ip_for_outbound_sockets=127.20.133.66
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34829
--builtin_ntp_servers=127.20.133.84:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:54:31.421571 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:31.410493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:44905 (local address 127.20.133.126:34829)
0927 13:54:31.410903 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:31.410907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:31.412115 (+ 1208us) server_negotiation.cc:407] Connection header received
0927 13:54:31.414261 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:31.414267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:31.414376 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:31.414489 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:31.415901 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:31.416763 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:31.418458 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:31.419103 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:31.419234 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:31.420670 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:31.420677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:31.420681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:31.420705 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:31.420754 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:31.420768 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:31.420855 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:31.421122 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 13:54:31.421225 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":219,"threads_started":1}
I20260927 13:54:31.422607 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:31.410993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34829 (local address 127.20.133.65:44905)
0927 13:54:31.411929 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:31.411966 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:31.413908 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:31.414796 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:31.414810 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:31.414945 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:31.415723 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:31.415745 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:31.416934 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:31.416939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:31.418300 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:31.418315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:31.419337 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:31.419343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:31.419565 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:31.420288 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:54:31.420290 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:31.420352 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:54:31.420469 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:31.420898 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:31.420908 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:31.420915 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 13:54:31.421097 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 13:54:31.421433 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":292,"threads_started":1}
I20260927 13:54:31.424310 8623 heartbeater.cc:347] Connected to a master server at 127.20.133.126:34829
I20260927 13:54:31.424716 8623 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:31.425552 8623 heartbeater.cc:511] Master 127.20.133.126:34829 requested a full tablet report, sending...
I20260927 13:54:31.427809 8437 ts_manager.cc:194] Registered new tserver with Master: 7d0e59c8d53c4747bd20ac6fd4112d06 (127.20.133.65:33137)
I20260927 13:54:31.429370 8437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.65:44905
W20260927 13:54:31.754196 8627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:31.754493 8627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:31.754606 8627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:31.762544 8627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:31.762693 8627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:31.762847 8627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.66
I20260927 13:54:31.772917 8627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.133.66
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.66
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:31.774250 8627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:31.776083 8627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:31.789687 8633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:31.789709 8635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:31.789651 8632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:31.790472 8627 server_base.cc:1034] running on GCE node
I20260927 13:54:31.791138 8627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:31.792117 8627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:31.793359 8627 hybrid_clock.cc:648] HybridClock initialized: now 1790517271793283 us; error 62 us; skew 500 ppm
I20260927 13:54:31.793696 8627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:31.796110 8627 webserver.cc:460] Webserver started at http://127.20.133.66:34431/ using document root <none> and password file <none>
I20260927 13:54:31.796871 8627 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:31.796967 8627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:31.797250 8627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:31.800192 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/data/instance:
uuid: "eb2dff347fe84618b61cbf92073123b0"
format_stamp: "Formatted at 2026-09-27 13:54:31 on dist-test-slave-q25r"
server_key: "0ef00fedadae2c3f196529b17d844788"
server_key_iv: "44b790daa2fa7b5cab9040d0e920243c"
server_key_version: "encryptionkey@0"
I20260927 13:54:31.801007 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance:
uuid: "eb2dff347fe84618b61cbf92073123b0"
format_stamp: "Formatted at 2026-09-27 13:54:31 on dist-test-slave-q25r"
server_key: "0ef00fedadae2c3f196529b17d844788"
server_key_iv: "44b790daa2fa7b5cab9040d0e920243c"
server_key_version: "encryptionkey@0"
I20260927 13:54:31.806740 8627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:54:31.810808 8641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:31.812433 8627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:54:31.812626 8627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/wal
uuid: "eb2dff347fe84618b61cbf92073123b0"
format_stamp: "Formatted at 2026-09-27 13:54:31 on dist-test-slave-q25r"
server_key: "0ef00fedadae2c3f196529b17d844788"
server_key_iv: "44b790daa2fa7b5cab9040d0e920243c"
server_key_version: "encryptionkey@0"
I20260927 13:54:31.812861 8627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:31.833681 8627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:31.834648 8627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:31.834960 8627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:31.835983 8627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:31.837567 8627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:31.837664 8627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:31.837776 8627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:31.837857 8627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:31.870755 8627 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.66:35183
I20260927 13:54:31.870839 8753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.66:35183 every 8 connection(s)
I20260927 13:54:31.872068 8627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/data/info.pb
I20260927 13:54:31.875742 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 8627
I20260927 13:54:31.875914 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/wal/instance
I20260927 13:54:31.876358 21013 external_mini_cluster.cc:1428] Setting key 24da25c787840615334f039b57ae6da2
I20260927 13:54:31.879317 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.133.67:0
--local_ip_for_outbound_sockets=127.20.133.67
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34829
--builtin_ntp_servers=127.20.133.84:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:54:31.885341 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:31.874705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.66:50211 (local address 127.20.133.126:34829)
0927 13:54:31.874873 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:31.874877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:31.876472 (+ 1595us) server_negotiation.cc:407] Connection header received
0927 13:54:31.878700 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:31.878706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:31.878789 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:31.878874 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:31.880074 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:31.880831 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:31.882434 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:31.882840 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:31.882949 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:31.884510 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:31.884516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:31.884519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:31.884541 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:31.884575 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:31.884587 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:31.884671 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:31.884904 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 13:54:31.885017 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:54:31.886416 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:31.875150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34829 (local address 127.20.133.66:50211)
0927 13:54:31.876256 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:31.876301 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:31.878373 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:31.879135 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:31.879144 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:31.879241 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:31.879899 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:31.879918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:31.880999 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:31.881003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:31.882275 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:31.882288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:31.883080 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:31.883088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:31.883341 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:31.884113 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:54:31.884118 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:31.884163 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:54:31.884258 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:31.884690 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:31.884702 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:31.884710 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:54:31.884876 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 13:54:31.885305 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":335,"threads_started":1}
I20260927 13:54:31.887866 8754 heartbeater.cc:347] Connected to a master server at 127.20.133.126:34829
I20260927 13:54:31.888165 8754 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:31.888973 8754 heartbeater.cc:511] Master 127.20.133.126:34829 requested a full tablet report, sending...
I20260927 13:54:31.890431 8437 ts_manager.cc:194] Registered new tserver with Master: eb2dff347fe84618b61cbf92073123b0 (127.20.133.66:35183)
I20260927 13:54:31.891247 8437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.66:50211
W20260927 13:54:32.211973 8757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:32.212291 8757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:32.212445 8757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:32.219970 8757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:32.220110 8757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:32.220273 8757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.133.67
I20260927 13:54:32.229744 8757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.133.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.133.67
--webserver_port=0
--tserver_master_addrs=127.20.133.126:34829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.133.67
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:32.231119 8757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:32.232908 8757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:32.246821 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:32.247062 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:32.247700 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:32.248117 8757 server_base.cc:1034] running on GCE node
I20260927 13:54:32.248814 8757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:32.249845 8757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:32.251052 8757 hybrid_clock.cc:648] HybridClock initialized: now 1790517272251009 us; error 41 us; skew 500 ppm
I20260927 13:54:32.251372 8757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:54:32.253823 8757 webserver.cc:460] Webserver started at http://127.20.133.67:33329/ using document root <none> and password file <none>
I20260927 13:54:32.254528 8757 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:32.254624 8757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:32.254937 8757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:32.258062 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/data/instance:
uuid: "3fec7a46b99f48abab2258976e6cfe5b"
format_stamp: "Formatted at 2026-09-27 13:54:32 on dist-test-slave-q25r"
server_key: "4c1b1f62f6e5caece76cb2081599bf17"
server_key_iv: "d6e5c553cb8e3dcc168986a9a49aeff2"
server_key_version: "encryptionkey@0"
I20260927 13:54:32.258872 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance:
uuid: "3fec7a46b99f48abab2258976e6cfe5b"
format_stamp: "Formatted at 2026-09-27 13:54:32 on dist-test-slave-q25r"
server_key: "4c1b1f62f6e5caece76cb2081599bf17"
server_key_iv: "d6e5c553cb8e3dcc168986a9a49aeff2"
server_key_version: "encryptionkey@0"
I20260927 13:54:32.264739 8757 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 13:54:32.268538 8771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:32.269877 8757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:54:32.270030 8757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/wal
uuid: "3fec7a46b99f48abab2258976e6cfe5b"
format_stamp: "Formatted at 2026-09-27 13:54:32 on dist-test-slave-q25r"
server_key: "4c1b1f62f6e5caece76cb2081599bf17"
server_key_iv: "d6e5c553cb8e3dcc168986a9a49aeff2"
server_key_version: "encryptionkey@0"
I20260927 13:54:32.270255 8757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:32.311956 8757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:32.312875 8757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:32.313169 8757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:32.314152 8757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:54:32.315601 8757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:54:32.315687 8757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:32.315765 8757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:54:32.315820 8757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:32.352046 8757 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.67:38617
I20260927 13:54:32.352152 8883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.67:38617 every 8 connection(s)
I20260927 13:54:32.353511 8757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/data/info.pb
I20260927 13:54:32.355394 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 8757
I20260927 13:54:32.355543 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/wal/instance
I20260927 13:54:32.355964 21013 external_mini_cluster.cc:1428] Setting key 66313548dccfe0c6cd4698223fb3953d
I20260927 13:54:32.367352 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.356918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.67:60255 (local address 127.20.133.126:34829)
0927 13:54:32.357175 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:32.357181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:32.358933 (+ 1752us) server_negotiation.cc:407] Connection header received
0927 13:54:32.360798 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:32.360805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:32.360892 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:32.360979 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:32.362171 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.362995 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.364577 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.365012 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.365123 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.366535 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:32.366540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:32.366543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:32.366563 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:32.366591 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:32.366603 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:32.366685 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:32.366904 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 13:54:32.367006 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 13:54:32.368414 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.357529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:34829 (local address 127.20.133.67:60255)
0927 13:54:32.358732 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:32.358788 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:32.360509 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:32.361230 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:32.361238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:32.361338 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:54:32.362001 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.362017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.363134 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.363137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.364439 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.364452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.365274 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.365281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.365507 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.366209 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:54:32.366211 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:54:32.366250 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:54:32.366337 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:54:32.366695 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:54:32.366705 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:54:32.366714 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 13:54:32.366872 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 13:54:32.367303 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":344,"threads_started":1}
I20260927 13:54:32.370317 8884 heartbeater.cc:347] Connected to a master server at 127.20.133.126:34829
I20260927 13:54:32.370721 8884 heartbeater.cc:464] Registering TS with master...
I20260927 13:54:32.371876 8884 heartbeater.cc:511] Master 127.20.133.126:34829 requested a full tablet report, sending...
I20260927 13:54:32.373689 8437 ts_manager.cc:194] Registered new tserver with Master: 3fec7a46b99f48abab2258976e6cfe5b (127.20.133.67:38617)
I20260927 13:54:32.374747 8437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.133.67:60255
I20260927 13:54:32.386606 21013 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:54:32.398401 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.391917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.20.133.126:34829)
0927 13:54:32.392100 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:32.392105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:32.392557 (+ 452us) server_negotiation.cc:407] Connection header received
0927 13:54:32.392722 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:32.392727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:32.392821 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:32.392947 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:32.393486 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.394234 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.395695 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.396121 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.396241 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.397655 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:32.397661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:32.397664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:32.397686 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:32.397729 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:32.397742 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:32.397841 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:32.398043 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 13:54:32.398139 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:54:32.403232 8437 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:54:32.410310 8437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:54:32.432765 8623 heartbeater.cc:503] Master 127.20.133.126:34829 was elected leader, sending a full tablet report...
I20260927 13:54:32.451825 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.424786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:38617 (local address 127.0.0.1:42900)
0927 13:54:32.442988 (+ 18202us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:32.443008 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:32.443166 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:32.443982 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:32.443991 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:32.444021 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:32.444404 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.444419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.446602 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.446609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.449102 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.449121 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.450855 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.450862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.451079 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.451130 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 13:54:32.451251 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 13:54:32.451480 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18044,"thread_start_us":17874,"threads_started":1}
I20260927 13:54:32.451867 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.424805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.20.133.67:38617)
0927 13:54:32.442650 (+ 17845us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:32.442659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:32.443317 (+ 658us) server_negotiation.cc:407] Connection header received
0927 13:54:32.443440 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:32.443449 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:32.443734 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:32.443926 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:32.444648 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.446390 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.449322 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.450642 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.450762 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.451061 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:32.451330 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 13:54:32.451505 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17647,"thread_start_us":17493,"threads_started":1}
I20260927 13:54:32.454576 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.424546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33137 (local address 127.0.0.1:58984)
0927 13:54:32.440934 (+ 16388us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:32.440998 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:32.441248 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:32.445174 (+ 3926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:32.445182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:32.445216 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:32.445734 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.445757 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.448143 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.448149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.451359 (+ 3210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.451376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.453665 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.453673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.453889 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.453926 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 13:54:32.454051 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 13:54:32.454234 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16079,"thread_start_us":16865,"threads_started":1}
I20260927 13:54:32.454797 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.425495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.20.133.65:33137)
0927 13:54:32.444081 (+ 18586us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:32.444093 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:32.444139 (+ 46us) server_negotiation.cc:407] Connection header received
0927 13:54:32.444272 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:32.444281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:32.444888 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:32.445975 (+ 1087us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:32.446315 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.447958 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.451606 (+ 3648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.453420 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.453570 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.453912 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:32.454145 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 13:54:32.454358 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18332,"thread_start_us":18138,"threads_started":1}
I20260927 13:54:32.456575 8819 tablet_service.cc:1467] Processing CreateTablet for tablet ab1269fc234c47029d3a64b447248ffd (DEFAULT_TABLE table=test-table [id=20ead91ba9e14c009910ce2d51e76363]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:32.458139 8819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1269fc234c47029d3a64b447248ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:32.459462 8558 tablet_service.cc:1467] Processing CreateTablet for tablet ab1269fc234c47029d3a64b447248ffd (DEFAULT_TABLE table=test-table [id=20ead91ba9e14c009910ce2d51e76363]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:32.461117 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.444406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:35183 (local address 127.0.0.1:33086)
0927 13:54:32.448312 (+ 3906us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:32.448380 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:32.448532 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:32.449393 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:32.449400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:32.449437 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:32.450127 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.450140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.452072 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.452077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.454152 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.454166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.456016 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.456023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.460398 (+ 4375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.460444 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 13:54:32.460564 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 13:54:32.460749 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3736,"thread_start_us":3539,"threads_started":1}
I20260927 13:54:32.461167 8558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1269fc234c47029d3a64b447248ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:32.461342 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.443536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.20.133.66:35183)
0927 13:54:32.447704 (+ 4168us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:32.447714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:32.448679 (+ 965us) server_negotiation.cc:407] Connection header received
0927 13:54:32.448815 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:32.448824 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:32.449159 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:32.449408 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:32.450366 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.451862 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.454400 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.455797 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.455936 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.456591 (+ 655us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:32.460713 (+ 4122us) server_negotiation.cc:299] Negotiation successful
0927 13:54:32.460966 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3957,"thread_start_us":3789,"threads_started":1}
I20260927 13:54:32.469401 8689 tablet_service.cc:1467] Processing CreateTablet for tablet ab1269fc234c47029d3a64b447248ffd (DEFAULT_TABLE table=test-table [id=20ead91ba9e14c009910ce2d51e76363]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:54:32.471118 8689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1269fc234c47029d3a64b447248ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:32.472704 8902 tablet_bootstrap.cc:492] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: Bootstrap starting.
I20260927 13:54:32.476630 8903 tablet_bootstrap.cc:492] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06: Bootstrap starting.
I20260927 13:54:32.478351 8902 tablet_bootstrap.cc:654] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:32.479612 8902 log.cc:824] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:32.481001 8903 tablet_bootstrap.cc:654] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:32.482481 8903 log.cc:824] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:32.482882 8902 tablet_bootstrap.cc:492] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: No bootstrap required, opened a new log
I20260927 13:54:32.483209 8902 ts_tablet_manager.cc:1397] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260927 13:54:32.486370 8903 tablet_bootstrap.cc:492] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06: No bootstrap required, opened a new log
I20260927 13:54:32.486487 8906 tablet_bootstrap.cc:492] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: Bootstrap starting.
I20260927 13:54:32.486855 8903 ts_tablet_manager.cc:1397] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260927 13:54:32.492318 8906 tablet_bootstrap.cc:654] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:32.493865 8906 log.cc:824] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:32.493847 8902 raft_consensus.cc:348] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33137 } } peers { permanent_uuid: "3fec7a46b99f48abab2258976e6cfe5b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 38617 } } peers { permanent_uuid: "eb2dff347fe84618b61cbf92073123b0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35183 } }
I20260927 13:54:32.494391 8902 raft_consensus.cc:374] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:32.494601 8902 raft_consensus.cc:729] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fec7a46b99f48abab2258976e6cfe5b, State: Initialized, Role: FOLLOWER
I20260927 13:54:32.495393 8902 consensus_queue.cc:260] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33137 } } peers { permanent_uuid: "3fec7a46b99f48abab2258976e6cfe5b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 38617 } } peers { permanent_uuid: "eb2dff347fe84618b61cbf92073123b0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35183 } }
I20260927 13:54:32.495620 8903 raft_consensus.cc:348] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33137 } } peers { permanent_uuid: "3fec7a46b99f48abab2258976e6cfe5b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 38617 } } peers { permanent_uuid: "eb2dff347fe84618b61cbf92073123b0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35183 } }
I20260927 13:54:32.496258 8903 raft_consensus.cc:374] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:32.496424 8903 raft_consensus.cc:729] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0e59c8d53c4747bd20ac6fd4112d06, State: Initialized, Role: FOLLOWER
I20260927 13:54:32.496659 8884 heartbeater.cc:503] Master 127.20.133.126:34829 was elected leader, sending a full tablet report...
I20260927 13:54:32.497412 8902 ts_tablet_manager.cc:1428] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260927 13:54:32.497318 8903 consensus_queue.cc:260] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33137 } } peers { permanent_uuid: "3fec7a46b99f48abab2258976e6cfe5b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 38617 } } peers { permanent_uuid: "eb2dff347fe84618b61cbf92073123b0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35183 } }
I20260927 13:54:32.497891 8906 tablet_bootstrap.cc:492] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: No bootstrap required, opened a new log
I20260927 13:54:32.498260 8906 ts_tablet_manager.cc:1397] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 13:54:32.499630 8903 ts_tablet_manager.cc:1428] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260927 13:54:32.508679 8906 raft_consensus.cc:348] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33137 } } peers { permanent_uuid: "3fec7a46b99f48abab2258976e6cfe5b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 38617 } } peers { permanent_uuid: "eb2dff347fe84618b61cbf92073123b0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35183 } }
I20260927 13:54:32.509106 8906 raft_consensus.cc:374] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:32.509217 8906 raft_consensus.cc:729] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2dff347fe84618b61cbf92073123b0, State: Initialized, Role: FOLLOWER
I20260927 13:54:32.509824 8906 consensus_queue.cc:260] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33137 } } peers { permanent_uuid: "3fec7a46b99f48abab2258976e6cfe5b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 38617 } } peers { permanent_uuid: "eb2dff347fe84618b61cbf92073123b0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35183 } }
I20260927 13:54:32.511132 8754 heartbeater.cc:503] Master 127.20.133.126:34829 was elected leader, sending a full tablet report...
I20260927 13:54:32.511518 8906 ts_tablet_manager.cc:1428] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
W20260927 13:54:32.605649 8885 tablet.cc:2369] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:54:32.624387 8755 tablet.cc:2369] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:54:32.660619 8624 tablet.cc:2369] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:54:32.829169 8909 raft_consensus.cc:482] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:54:32.829607 8909 raft_consensus.cc:504] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33137 } } peers { permanent_uuid: "3fec7a46b99f48abab2258976e6cfe5b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 38617 } } peers { permanent_uuid: "eb2dff347fe84618b61cbf92073123b0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35183 } }
I20260927 13:54:32.831710 8909 leader_election.cc:288] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fec7a46b99f48abab2258976e6cfe5b (127.20.133.67:38617), eb2dff347fe84618b61cbf92073123b0 (127.20.133.66:35183)
I20260927 13:54:32.840181 8911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.832086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.67:38617 (local address 127.20.133.65:57469)
0927 13:54:32.832937 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:32.832969 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:32.833216 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:32.833886 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:32.833891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:32.833924 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:32.834481 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.834500 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.835789 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.835795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.837922 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.837938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.839349 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.839357 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.839537 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.839579 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 13:54:32.839675 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 13:54:32.839817 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":120,"thread_start_us":481,"threads_started":1}
I20260927 13:54:32.840430 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.832294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:57469 (local address 127.20.133.67:38617)
0927 13:54:32.832636 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:32.832644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:32.833144 (+ 500us) server_negotiation.cc:407] Connection header received
0927 13:54:32.833478 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:32.833485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:32.833581 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:32.833655 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:32.834701 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.835616 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.838197 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.839048 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.839209 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.839268 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:32.839836 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 13:54:32.840045 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 13:54:32.840659 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.833411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.133.65:50703 (local address 127.20.133.66:35183)
0927 13:54:32.833651 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:32.833657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:32.833939 (+ 282us) server_negotiation.cc:407] Connection header received
0927 13:54:32.834155 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:32.834161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:32.834268 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:32.834396 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:54:32.835166 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.836024 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.838397 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.839330 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.839505 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.839584 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:32.840197 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 13:54:32.840399 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 13:54:32.841118 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.833018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.66:35183 (local address 127.20.133.65:50703)
0927 13:54:32.833792 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:32.833812 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:54:32.833999 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:54:32.834502 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:54:32.834506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:54:32.834536 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:54:32.834964 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.834980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.836227 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.836232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.838132 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:54:32.838156 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.839601 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.839608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:54:32.839919 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.839987 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 13:54:32.840145 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 13:54:32.840323 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":74,"thread_start_us":395,"threads_started":1}
I20260927 13:54:32.841897 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1269fc234c47029d3a64b447248ffd" candidate_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fec7a46b99f48abab2258976e6cfe5b" is_pre_election: true
I20260927 13:54:32.842633 8839 raft_consensus.cc:2454] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d0e59c8d53c4747bd20ac6fd4112d06 in term 0.
I20260927 13:54:32.842464 8709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1269fc234c47029d3a64b447248ffd" candidate_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2dff347fe84618b61cbf92073123b0" is_pre_election: true
I20260927 13:54:32.843036 8709 raft_consensus.cc:2454] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d0e59c8d53c4747bd20ac6fd4112d06 in term 0.
I20260927 13:54:32.843501 8514 leader_election.cc:302] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fec7a46b99f48abab2258976e6cfe5b, 7d0e59c8d53c4747bd20ac6fd4112d06; no voters:
I20260927 13:54:32.843967 8909 raft_consensus.cc:2781] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:54:32.844128 8909 raft_consensus.cc:482] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:54:32.844231 8909 raft_consensus.cc:3037] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:32.846962 8909 raft_consensus.cc:504] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33137 } } peers { permanent_uuid: "3fec7a46b99f48abab2258976e6cfe5b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 38617 } } peers { permanent_uuid: "eb2dff347fe84618b61cbf92073123b0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35183 } }
I20260927 13:54:32.847708 8909 leader_election.cc:288] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [CANDIDATE]: Term 1 election: Requested vote from peers 3fec7a46b99f48abab2258976e6cfe5b (127.20.133.67:38617), eb2dff347fe84618b61cbf92073123b0 (127.20.133.66:35183)
I20260927 13:54:32.848459 8709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1269fc234c47029d3a64b447248ffd" candidate_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2dff347fe84618b61cbf92073123b0"
I20260927 13:54:32.848577 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1269fc234c47029d3a64b447248ffd" candidate_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fec7a46b99f48abab2258976e6cfe5b"
I20260927 13:54:32.848677 8709 raft_consensus.cc:3037] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:32.848901 8839 raft_consensus.cc:3037] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:32.851186 8709 raft_consensus.cc:2454] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d0e59c8d53c4747bd20ac6fd4112d06 in term 1.
I20260927 13:54:32.851393 8839 raft_consensus.cc:2454] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d0e59c8d53c4747bd20ac6fd4112d06 in term 1.
I20260927 13:54:32.851768 8514 leader_election.cc:302] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d0e59c8d53c4747bd20ac6fd4112d06, eb2dff347fe84618b61cbf92073123b0; no voters:
I20260927 13:54:32.852283 8909 raft_consensus.cc:2781] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:32.852864 8909 raft_consensus.cc:686] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 1 LEADER]: Becoming Leader. State: Replica: 7d0e59c8d53c4747bd20ac6fd4112d06, State: Running, Role: LEADER
I20260927 13:54:32.853441 8909 consensus_queue.cc:237] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33137 } } peers { permanent_uuid: "3fec7a46b99f48abab2258976e6cfe5b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 38617 } } peers { permanent_uuid: "eb2dff347fe84618b61cbf92073123b0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35183 } }
I20260927 13:54:32.859064 8436 catalog_manager.cc:5495] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d0e59c8d53c4747bd20ac6fd4112d06 (127.20.133.65). New cstate: current_term: 1 leader_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0e59c8d53c4747bd20ac6fd4112d06" member_type: VOTER last_known_addr { host: "127.20.133.65" port: 33137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fec7a46b99f48abab2258976e6cfe5b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 38617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb2dff347fe84618b61cbf92073123b0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35183 } health_report { overall_health: UNKNOWN } } }
I20260927 13:54:32.902989 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.896760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.20.133.65:33137)
0927 13:54:32.897038 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:32.897044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:32.897139 (+ 95us) server_negotiation.cc:407] Connection header received
0927 13:54:32.897451 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:32.897456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:32.897590 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:32.897763 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:54:32.898196 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.898983 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.900490 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.900943 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.901057 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.901624 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:54:32.902324 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:54:32.902473 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:32.902582 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 13:54:32.902679 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 13:54:32.913658 8709 raft_consensus.cc:1261] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0 [term 1 FOLLOWER]: Refusing update from remote peer 7d0e59c8d53c4747bd20ac6fd4112d06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:32.913718 8839 raft_consensus.cc:1261] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b [term 1 FOLLOWER]: Refusing update from remote peer 7d0e59c8d53c4747bd20ac6fd4112d06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:54:32.914981 8913 consensus_queue.cc:1035] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2dff347fe84618b61cbf92073123b0" member_type: VOTER last_known_addr { host: "127.20.133.66" port: 35183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:32.915534 8909 consensus_queue.cc:1035] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fec7a46b99f48abab2258976e6cfe5b" member_type: VOTER last_known_addr { host: "127.20.133.67" port: 38617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:54:32.944741 8437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:32.945101 8437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:54:32.949826 8437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f9ddb07643548ad85fb1411eb792bbd: Sending DeleteTablet for 3 replicas of tablet ab1269fc234c47029d3a64b447248ffd
I20260927 13:54:32.951489 8558 tablet_service.cc:1514] Processing DeleteTablet for tablet ab1269fc234c47029d3a64b447248ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:32 UTC) from {username='slave'} at 127.0.0.1:58984
I20260927 13:54:32.951537 8819 tablet_service.cc:1514] Processing DeleteTablet for tablet ab1269fc234c47029d3a64b447248ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:32 UTC) from {username='slave'} at 127.0.0.1:42900
I20260927 13:54:32.951913 8689 tablet_service.cc:1514] Processing DeleteTablet for tablet ab1269fc234c47029d3a64b447248ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:54:32 UTC) from {username='slave'} at 127.0.0.1:33086
I20260927 13:54:32.954115 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 8496
I20260927 13:54:32.962750 8928 tablet_replica.cc:330] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: stopping tablet replica
I20260927 13:54:32.963385 8926 tablet_replica.cc:330] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: stopping tablet replica
I20260927 13:54:32.963605 8928 raft_consensus.cc:2229] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:32.964118 8926 raft_consensus.cc:2229] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:54:32.964376 8928 raft_consensus.cc:2258] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:32.964843 8926 raft_consensus.cc:2258] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:32.967140 8928 ts_tablet_manager.cc:1905] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:32.972558 8926 ts_tablet_manager.cc:1905] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:32.974210 8927 tablet_replica.cc:330] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06: stopping tablet replica
I20260927 13:54:32.974988 8927 raft_consensus.cc:2229] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:54:32.975638 8928 ts_tablet_manager.cc:1918] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:32.975966 8928 log.cc:1196] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-1/wal/wals/ab1269fc234c47029d3a64b447248ffd
I20260927 13:54:32.975979 8927 raft_consensus.cc:2258] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:54:32.977316 8928 ts_tablet_manager.cc:1939] T ab1269fc234c47029d3a64b447248ffd P eb2dff347fe84618b61cbf92073123b0: Deleting consensus metadata
I20260927 13:54:32.978924 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:32.955387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.20.133.65:33137)
0927 13:54:32.955602 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:32.955610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:32.962334 (+ 6724us) server_negotiation.cc:407] Connection header received
0927 13:54:32.962571 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:32.962577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:32.962697 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:32.962821 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:32.963673 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.964978 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.967388 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:32.968199 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:32.968321 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:32.976242 (+ 7921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:32.976255 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:32.976276 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:32.977300 (+ 1024us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:32.977376 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:32.977404 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:32.977522 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:32.978350 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 13:54:32.978488 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 13:54:32.979120 8927 ts_tablet_manager.cc:1905] T ab1269fc234c47029d3a64b447248ffd P 7d0e59c8d53c4747bd20ac6fd4112d06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:54:32.979847 8618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:32.982169 8424 catalog_manager.cc:4841] TS eb2dff347fe84618b61cbf92073123b0 (127.20.133.66:35183): tablet ab1269fc234c47029d3a64b447248ffd (table test-table [id=20ead91ba9e14c009910ce2d51e76363]) successfully deleted
I20260927 13:54:32.982633 8926 ts_tablet_manager.cc:1918] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:54:32.982913 8926 log.cc:1196] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: Deleting WAL directory at /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516948056776-21013-0/minicluster-data/ts-2/wal/wals/ab1269fc234c47029d3a64b447248ffd
I20260927 13:54:32.983615 8926 ts_tablet_manager.cc:1939] T ab1269fc234c47029d3a64b447248ffd P 3fec7a46b99f48abab2258976e6cfe5b: Deleting consensus metadata
I20260927 13:54:32.986063 8424 catalog_manager.cc:4841] TS 3fec7a46b99f48abab2258976e6cfe5b (127.20.133.67:38617): tablet ab1269fc234c47029d3a64b447248ffd (table test-table [id=20ead91ba9e14c009910ce2d51e76363]) successfully deleted
I20260927 13:54:33.081327 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 8496
I20260927 13:54:33.097637 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 8627
W20260927 13:54:33.097828 8423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.133.65:33137 (error 108)
I20260927 13:54:33.100811 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:33.099533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33137 (local address 127.0.0.1:59006)
0927 13:54:33.100143 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:33.100365 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":272,"threads_started":1}
W20260927 13:54:33.101152 8423 catalog_manager.cc:4584] TS 7d0e59c8d53c4747bd20ac6fd4112d06 (127.20.133.65:33137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab1269fc234c47029d3a64b447248ffd: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33137: connect: Connection refused (error 111)
I20260927 13:54:33.106148 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:33.098571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.20.133.66:35183)
0927 13:54:33.098757 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:33.098762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:33.098785 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:54:33.099098 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:33.099104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:33.099209 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:33.099316 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:33.099993 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:33.101231 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:33.102729 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:33.103166 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:33.103299 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:33.104559 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:33.104569 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:33.104587 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:33.105238 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:33.105298 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:33.105318 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:33.105426 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:33.105651 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 13:54:33.105764 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 13:54:33.107087 8749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:33.155068 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:33.154333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33137 (local address 127.0.0.1:59018)
0927 13:54:33.154647 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:33.154808 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 13:54:33.156323 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:33.155885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33137 (local address 127.0.0.1:59024)
0927 13:54:33.156044 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:33.156137 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 13:54:33.207415 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 8627
I20260927 13:54:33.217288 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:33.216628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33137 (local address 127.0.0.1:59032)
0927 13:54:33.216946 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:33.217078 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 13:54:33.218833 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:33.218311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33137 (local address 127.0.0.1:59044)
0927 13:54:33.218499 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:33.218606 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 13:54:33.222952 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 8757
I20260927 13:54:33.231078 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:33.224051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.20.133.67:38617)
0927 13:54:33.224233 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:33.224238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:33.224259 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:54:33.224420 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:33.224424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:33.224538 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:33.224641 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:33.225195 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:33.225966 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:33.227596 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:33.228033 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:33.228135 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:33.229582 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:33.229590 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:33.229605 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:33.230174 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:33.230251 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:33.230272 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:33.230381 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:33.230612 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 13:54:33.230724 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:54:33.231910 8879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:33.299889 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:33.299111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33137 (local address 127.0.0.1:59058)
0927 13:54:33.299468 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:33.299626 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260927 13:54:33.301397 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:33.300849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.65:33137 (local address 127.0.0.1:59060)
0927 13:54:33.301101 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 13:54:33.301201 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.65:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260927 13:54:33.332304 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 8757
I20260927 13:54:33.347581 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 8406
I20260927 13:54:33.349063 8467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:33.472856 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 8406
2026-09-27T13:54:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3283 ms)
[----------] 6 tests from EncryptionPolicyTest (22336 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:33 dist-test-slave-q25r krb5kdc[8938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:33 dist-test-slave-q25r krb5kdc[8938](info): set up 2 sockets
Sep 27 13:54:33 dist-test-slave-q25r krb5kdc[8938](info): commencing operation
krb5kdc: starting...
W20260927 13:54:34.524742 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.008s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:34 dist-test-slave-q25r krb5kdc[8938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:34.784523 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:35.131865 8951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:35.132210 8951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:35.132375 8951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:35.140322 8951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:35.140522 8951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:35.140609 8951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:35.140689 8951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:35.140750 8951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:35.151440 8951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:35.152868 8951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:35.154206 8951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:35.167368 8956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:35.167459 8957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:35.167469 8959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:35.169064 8951 server_base.cc:1034] running on GCE node
I20260927 13:54:35.170032 8951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:35.171721 8951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:35.172981 8951 hybrid_clock.cc:648] HybridClock initialized: now 1790517275172929 us; error 53 us; skew 500 ppm
I20260927 13:54:35.173398 8951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:35 dist-test-slave-q25r krb5kdc[8938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:35.183221 8951 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:35.185091 8951 webserver.cc:460] Webserver started at http://127.0.0.1:36993/ using document root <none> and password file <none>
I20260927 13:54:35.186019 8951 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:35.186160 8951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:35.186436 8951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:35.190308 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "bcf02e2dd9f844f0a746f80f0ece8bde"
format_stamp: "Formatted at 2026-09-27 13:54:35 on dist-test-slave-q25r"
server_key: "8525322da4ebd7c2569f0e92b3c4851f"
server_key_iv: "86f6e7e89415f342cf962f1ede880037"
server_key_version: "encryptionkey@0"
I20260927 13:54:35.191169 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "bcf02e2dd9f844f0a746f80f0ece8bde"
format_stamp: "Formatted at 2026-09-27 13:54:35 on dist-test-slave-q25r"
server_key: "8525322da4ebd7c2569f0e92b3c4851f"
server_key_iv: "86f6e7e89415f342cf962f1ede880037"
server_key_version: "encryptionkey@0"
I20260927 13:54:35.197247 8951 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 13:54:35.203605 8966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:35.205652 8951 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:54:35.205880 8951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "bcf02e2dd9f844f0a746f80f0ece8bde"
format_stamp: "Formatted at 2026-09-27 13:54:35 on dist-test-slave-q25r"
server_key: "8525322da4ebd7c2569f0e92b3c4851f"
server_key_iv: "86f6e7e89415f342cf962f1ede880037"
server_key_version: "encryptionkey@0"
I20260927 13:54:35.206168 8951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:35.234552 8951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:35.240479 8951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:35.240813 8951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:35.260509 8951 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44017
I20260927 13:54:35.260512 9017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44017 every 8 connection(s)
I20260927 13:54:35.261922 8951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:35.265805 9018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:35.270941 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 8951
I20260927 13:54:35.271143 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:35.271759 21013 external_mini_cluster.cc:1428] Setting key af0f18078ec1fde87cb524b899eeaf35
I20260927 13:54:35.277964 9018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde: Bootstrap starting.
I20260927 13:54:35.281302 9018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:35.282374 9018 log.cc:824] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:35.285368 9018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde: No bootstrap required, opened a new log
Sep 27 13:54:35 dist-test-slave-q25r krb5kdc[8938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 13:54:35.293746 9018 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf02e2dd9f844f0a746f80f0ece8bde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44017 } }
I20260927 13:54:35.294166 9018 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:35.294283 9018 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf02e2dd9f844f0a746f80f0ece8bde, State: Initialized, Role: FOLLOWER
I20260927 13:54:35.294924 9018 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf02e2dd9f844f0a746f80f0ece8bde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44017 } }
I20260927 13:54:35.295195 9018 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:35.295328 9018 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:35.295495 9018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:35.295673 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:35.273784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35585 (local address 127.0.0.1:44017)
0927 13:54:35.274842 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:35.274856 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:35.274905 (+ 49us) server_negotiation.cc:407] Connection header received
0927 13:54:35.276826 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:35.276851 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:35.277297 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:35.277673 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:35.279178 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:35.280585 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:35.281936 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:35.282672 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:35.282837 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:35.287252 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:35.287260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:35.287277 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:35.287325 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:35.290789 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:35.291811 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:35.291818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:35.291829 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:35.292041 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:35.292607 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:35.292611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:35.292613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:35.293228 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:35.293541 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:35.294085 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 13:54:35.294495 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":396,"threads_started":1}
I20260927 13:54:35.298472 9018 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf02e2dd9f844f0a746f80f0ece8bde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44017 } }
I20260927 13:54:35.299045 9018 leader_election.cc:302] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcf02e2dd9f844f0a746f80f0ece8bde; no voters:
I20260927 13:54:35.299705 9018 leader_election.cc:288] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:35.299872 9023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:35.300172 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:35.301786 9018 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:35.307101 9023 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [term 1 LEADER]: Becoming Leader. State: Replica: bcf02e2dd9f844f0a746f80f0ece8bde, State: Running, Role: LEADER
I20260927 13:54:35.307977 9023 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf02e2dd9f844f0a746f80f0ece8bde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44017 } }
I20260927 13:54:35.311667 9024 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcf02e2dd9f844f0a746f80f0ece8bde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf02e2dd9f844f0a746f80f0ece8bde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44017 } } }
I20260927 13:54:35.312151 9024 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:35.312739 9025 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcf02e2dd9f844f0a746f80f0ece8bde. Latest consensus state: current_term: 1 leader_uuid: "bcf02e2dd9f844f0a746f80f0ece8bde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf02e2dd9f844f0a746f80f0ece8bde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44017 } } }
I20260927 13:54:35.313046 9025 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:35.316179 9038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:35.321560 9038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:35.336117 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:35.312171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37345 (local address 127.0.0.1:44017)
0927 13:54:35.312446 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:35.312452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:35.314897 (+ 2445us) server_negotiation.cc:407] Connection header received
0927 13:54:35.315102 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:35.315108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:35.315204 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:35.315335 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:35.317569 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:35.318779 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:35.320645 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:35.321381 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:35.321532 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:35.325849 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:35.325863 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:35.325870 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:35.325929 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:35.330858 (+ 4929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:35.332251 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:35.332258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:35.332262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:35.332499 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:35.333218 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:35.333224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:35.333227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:35.333705 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:35.334003 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:35.335354 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0927 13:54:35.335575 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 13:54:35.347826 9038 catalog_manager.cc:1347] Generated new cluster ID: 7b430c726a524f00ae0c5d8015a19685
I20260927 13:54:35.347965 9038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:35.365338 9038 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:35.366694 9038 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:35.381793 9038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcf02e2dd9f844f0a746f80f0ece8bde: Generated new TSK 0
I20260927 13:54:35.383297 9038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:35.394878 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 8951
I20260927 13:54:35.396505 9013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:35.497401 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 8951
W20260927 13:54:35.512463 8936 connection.cc:535] client connection to 127.0.0.1:44017 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44017): Connection reset by peer (error 104)
2026-09-27T13:54:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:35 dist-test-slave-q25r krb5kdc[9050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:35 dist-test-slave-q25r krb5kdc[9050](info): set up 2 sockets
Sep 27 13:54:35 dist-test-slave-q25r krb5kdc[9050](info): commencing operation
krb5kdc: starting...
W20260927 13:54:36.528777 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:36 dist-test-slave-q25r krb5kdc[9050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:36.823583 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:37.166579 9063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:37.166898 9063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:37.167023 9063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:37.174542 9063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:37.174677 9063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:37.174727 9063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:37.174815 9063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:37.174885 9063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:37.185093 9063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:37.186457 9063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:37.187736 9063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:37.200831 9069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:37.200981 9071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:37.201143 9068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:37.202685 9063 server_base.cc:1034] running on GCE node
I20260927 13:54:37.203879 9063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:37.206178 9063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:37.207504 9063 hybrid_clock.cc:648] HybridClock initialized: now 1790517277207355 us; error 123 us; skew 500 ppm
I20260927 13:54:37.208069 9063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:37 dist-test-slave-q25r krb5kdc[9050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:37.212288 9063 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:37.214576 9063 webserver.cc:460] Webserver started at http://127.0.0.1:32827/ using document root <none> and password file <none>
I20260927 13:54:37.215541 9063 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:37.215699 9063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:37.216085 9063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:37.220188 9063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "eeec1d254b5947738601af3a264cd7d1"
format_stamp: "Formatted at 2026-09-27 13:54:37 on dist-test-slave-q25r"
server_key: "6b005585fd69b546af32c46eb7029d0d"
server_key_iv: "f3cb1088d1631c205f341569abdd36bd"
server_key_version: "encryptionkey@0"
I20260927 13:54:37.221262 9063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "eeec1d254b5947738601af3a264cd7d1"
format_stamp: "Formatted at 2026-09-27 13:54:37 on dist-test-slave-q25r"
server_key: "6b005585fd69b546af32c46eb7029d0d"
server_key_iv: "f3cb1088d1631c205f341569abdd36bd"
server_key_version: "encryptionkey@0"
I20260927 13:54:37.228663 9063 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 13:54:37.234133 9078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:37.236203 9063 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260927 13:54:37.236528 9063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "eeec1d254b5947738601af3a264cd7d1"
format_stamp: "Formatted at 2026-09-27 13:54:37 on dist-test-slave-q25r"
server_key: "6b005585fd69b546af32c46eb7029d0d"
server_key_iv: "f3cb1088d1631c205f341569abdd36bd"
server_key_version: "encryptionkey@0"
I20260927 13:54:37.236805 9063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:37.259528 9063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:37.264603 9063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:37.264930 9063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:37.286080 9063 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36395
I20260927 13:54:37.286129 9129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36395 every 8 connection(s)
I20260927 13:54:37.287995 9063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:37.292537 9130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:37.295495 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 9063
I20260927 13:54:37.295682 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:37.296159 21013 external_mini_cluster.cc:1428] Setting key 412a7fafd7439f6c8518ee449d28b727
I20260927 13:54:37.305979 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1: Bootstrap starting.
I20260927 13:54:37.309649 9130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:37.310881 9130 log.cc:824] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:37.312878 9133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:37.298425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49023 (local address 127.0.0.1:36395)
0927 13:54:37.299456 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:37.299475 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:37.299531 (+ 56us) server_negotiation.cc:407] Connection header received
0927 13:54:37.301817 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:37.301858 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:37.302509 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:37.303020 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:37.303769 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:37.305330 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:37.306876 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:37.307744 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:37.307979 (+ 235us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:37.309528 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:37.309542 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:37.309564 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:37.310359 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:37.310466 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:37.310499 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:37.310668 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:37.311017 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 13:54:37.311249 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":449,"threads_started":1}
I20260927 13:54:37.314872 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1: No bootstrap required, opened a new log
I20260927 13:54:37.318243 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:37.325572 9130 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeec1d254b5947738601af3a264cd7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } }
I20260927 13:54:37.326129 9130 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:37.326280 9130 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeec1d254b5947738601af3a264cd7d1, State: Initialized, Role: FOLLOWER
I20260927 13:54:37.326929 9130 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeec1d254b5947738601af3a264cd7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } }
I20260927 13:54:37.327184 9130 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:37.327332 9130 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:37.327500 9130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:37.330587 9130 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeec1d254b5947738601af3a264cd7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } }
I20260927 13:54:37.331282 9130 leader_election.cc:302] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeec1d254b5947738601af3a264cd7d1; no voters:
I20260927 13:54:37.332058 9130 leader_election.cc:288] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:37.332553 9133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:37.324102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47641 (local address 127.0.0.1:36395)
0927 13:54:37.324528 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:37.324536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:37.325024 (+ 488us) server_negotiation.cc:407] Connection header received
0927 13:54:37.325246 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:37.325251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:37.325342 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:37.325460 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:37.326202 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:37.327317 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:37.329369 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:37.329915 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:37.330026 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:37.331443 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:37.331450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:37.331454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:37.331484 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:37.331567 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:37.331588 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:37.331683 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:37.331959 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 13:54:37.332033 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 13:54:37.333722 9140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:37.335464 9140 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [term 1 LEADER]: Becoming Leader. State: Replica: eeec1d254b5947738601af3a264cd7d1, State: Running, Role: LEADER
I20260927 13:54:37.335727 9130 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:37.336131 9140 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeec1d254b5947738601af3a264cd7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } }
I20260927 13:54:37.339432 9142 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeec1d254b5947738601af3a264cd7d1. Latest consensus state: current_term: 1 leader_uuid: "eeec1d254b5947738601af3a264cd7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeec1d254b5947738601af3a264cd7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } } }
I20260927 13:54:37.339919 9142 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:37.340481 9141 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeec1d254b5947738601af3a264cd7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeec1d254b5947738601af3a264cd7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } } }
I20260927 13:54:37.340823 9141 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:37.341351 9148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:37.349479 9148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:37.361272 9148 catalog_manager.cc:1347] Generated new cluster ID: 6da32917db4948509a171ae5bd58057e
I20260927 13:54:37.361451 9148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:37.401527 9148 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:37.403245 9148 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:37.418012 9148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeec1d254b5947738601af3a264cd7d1: Generated new TSK 0
I20260927 13:54:37.418934 9148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:37.432488 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9063
W20260927 13:54:37.434305 9125 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49023
W20260927 13:54:37.435400 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9063: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:54:37.435644 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 9063
W20260927 13:54:37.450748 9048 connection.cc:535] client connection to 127.0.0.1:36395 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36395): Connection reset by peer (error 104)
2026-09-27T13:54:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:37 dist-test-slave-q25r krb5kdc[9161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:37 dist-test-slave-q25r krb5kdc[9161](info): set up 2 sockets
Sep 27 13:54:37 dist-test-slave-q25r krb5kdc[9161](info): commencing operation
krb5kdc: starting...
W20260927 13:54:38.468784 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:38 dist-test-slave-q25r krb5kdc[9161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:38.765065 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:39.156673 9174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:39.157086 9174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:39.157235 9174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:39.165562 9174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:39.165719 9174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:39.165807 9174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:39.165903 9174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:39.165968 9174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:39.177492 9174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:39.179261 9174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:39.180987 9174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:39.195122 9182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:39.195238 9180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:39.195228 9179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:39.198241 9174 server_base.cc:1034] running on GCE node
I20260927 13:54:39.199303 9174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:39.201254 9174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:39.202502 9174 hybrid_clock.cc:648] HybridClock initialized: now 1790517279202446 us; error 46 us; skew 500 ppm
I20260927 13:54:39.202935 9174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:39 dist-test-slave-q25r krb5kdc[9161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:39.212905 9174 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:39.215477 9174 webserver.cc:460] Webserver started at http://127.0.0.1:44193/ using document root <none> and password file <none>
I20260927 13:54:39.216655 9174 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:39.216782 9174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:39.217104 9174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:39.220819 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "f80fbd96b8c844f79b0cb2ef30d37b89"
format_stamp: "Formatted at 2026-09-27 13:54:39 on dist-test-slave-q25r"
server_key: "bef0a3c1fc4129b53765a56369fb519e"
server_key_iv: "6a7ba50560be6513d03bf6e01ae39e6d"
server_key_version: "encryptionkey@0"
I20260927 13:54:39.221674 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "f80fbd96b8c844f79b0cb2ef30d37b89"
format_stamp: "Formatted at 2026-09-27 13:54:39 on dist-test-slave-q25r"
server_key: "bef0a3c1fc4129b53765a56369fb519e"
server_key_iv: "6a7ba50560be6513d03bf6e01ae39e6d"
server_key_version: "encryptionkey@0"
I20260927 13:54:39.228273 9174 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 13:54:39.233246 9189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:39.235121 9174 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:54:39.235325 9174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "f80fbd96b8c844f79b0cb2ef30d37b89"
format_stamp: "Formatted at 2026-09-27 13:54:39 on dist-test-slave-q25r"
server_key: "bef0a3c1fc4129b53765a56369fb519e"
server_key_iv: "6a7ba50560be6513d03bf6e01ae39e6d"
server_key_version: "encryptionkey@0"
I20260927 13:54:39.235569 9174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:39.242592 9174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:39.247704 9174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:39.247998 9174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:39.269582 9174 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40169
I20260927 13:54:39.269599 9240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40169 every 8 connection(s)
I20260927 13:54:39.271487 9174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:39.276368 9241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:39.276974 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 9174
I20260927 13:54:39.277177 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:39.277748 21013 external_mini_cluster.cc:1428] Setting key 94da89ebd66b039f1d4f8f4943d17bb4
I20260927 13:54:39.288283 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:39.280097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46933 (local address 127.0.0.1:40169)
0927 13:54:39.281357 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:39.281373 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:39.281438 (+ 65us) server_negotiation.cc:407] Connection header received
0927 13:54:39.283614 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:39.283651 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:39.284227 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:39.284761 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:39.286200 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:39.286208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:39.286225 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:39.286250 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:39.286331 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:39.286362 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:39.286576 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:39.286707 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 13:54:39.286969 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":973,"thread_start_us":652,"threads_started":1}
I20260927 13:54:39.292483 9241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89: Bootstrap starting.
I20260927 13:54:39.293226 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:39.296321 9241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:39.297734 9241 log.cc:824] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:39.301467 9241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89: No bootstrap required, opened a new log
I20260927 13:54:39.301795 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:39.298976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47075 (local address 127.0.0.1:40169)
0927 13:54:39.299196 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:39.299202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:39.299514 (+ 312us) server_negotiation.cc:407] Connection header received
0927 13:54:39.299693 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:39.299699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:39.299768 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:39.299922 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:39.300993 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:39.300998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:39.301002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:39.301023 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:39.301077 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:39.301098 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:39.301258 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:39.301432 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 13:54:39.301509 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 13:54:39.309491 9241 raft_consensus.cc:348] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80fbd96b8c844f79b0cb2ef30d37b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260927 13:54:39.309966 9241 raft_consensus.cc:374] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:39.310089 9241 raft_consensus.cc:729] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f80fbd96b8c844f79b0cb2ef30d37b89, State: Initialized, Role: FOLLOWER
I20260927 13:54:39.310837 9241 consensus_queue.cc:260] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80fbd96b8c844f79b0cb2ef30d37b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260927 13:54:39.311094 9241 raft_consensus.cc:388] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:39.311235 9241 raft_consensus.cc:482] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:39.311398 9241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:39.314248 9241 raft_consensus.cc:504] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80fbd96b8c844f79b0cb2ef30d37b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260927 13:54:39.314844 9241 leader_election.cc:302] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f80fbd96b8c844f79b0cb2ef30d37b89; no voters:
I20260927 13:54:39.315513 9241 leader_election.cc:288] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:39.315754 9251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:39.317003 9251 raft_consensus.cc:686] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [term 1 LEADER]: Becoming Leader. State: Replica: f80fbd96b8c844f79b0cb2ef30d37b89, State: Running, Role: LEADER
I20260927 13:54:39.317771 9251 consensus_queue.cc:237] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80fbd96b8c844f79b0cb2ef30d37b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260927 13:54:39.318166 9241 sys_catalog.cc:564] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:39.321197 9253 sys_catalog.cc:455] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f80fbd96b8c844f79b0cb2ef30d37b89. Latest consensus state: current_term: 1 leader_uuid: "f80fbd96b8c844f79b0cb2ef30d37b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80fbd96b8c844f79b0cb2ef30d37b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } } }
I20260927 13:54:39.321292 9252 sys_catalog.cc:455] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f80fbd96b8c844f79b0cb2ef30d37b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f80fbd96b8c844f79b0cb2ef30d37b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } } }
I20260927 13:54:39.321712 9253 sys_catalog.cc:458] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:39.321787 9252 sys_catalog.cc:458] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:39.324467 9257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:39.330883 9257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:39.342537 9257 catalog_manager.cc:1347] Generated new cluster ID: ed9a266f821d49809a510d5f605359a6
I20260927 13:54:39.342690 9257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:39.368834 9257 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:39.369442 9257 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:39.376120 9257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f80fbd96b8c844f79b0cb2ef30d37b89: Generated new TSK 0
I20260927 13:54:39.377252 9257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:39.391074 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9174
W20260927 13:54:39.392650 9236 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46933
W20260927 13:54:39.393700 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9174: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:54:39.393945 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 9174
2026-09-27T13:54:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:39 dist-test-slave-q25r krb5kdc[9272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:39 dist-test-slave-q25r krb5kdc[9272](info): set up 2 sockets
Sep 27 13:54:39 dist-test-slave-q25r krb5kdc[9272](info): commencing operation
krb5kdc: starting...
W20260927 13:54:40.436571 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:40 dist-test-slave-q25r krb5kdc[9272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:40.731799 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:41.107419 9285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:41.107766 9285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:41.107894 9285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:41.115679 9285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:41.115825 9285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:41.115878 9285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:41.115944 9285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:41.116032 9285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:41.126449 9285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:41.127825 9285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:41.129195 9285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:41.141813 9290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:41.142051 9291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:41.142190 9293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:41.143441 9285 server_base.cc:1034] running on GCE node
I20260927 13:54:41.144585 9285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:41.146560 9285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:41.147832 9285 hybrid_clock.cc:648] HybridClock initialized: now 1790517281147725 us; error 106 us; skew 500 ppm
I20260927 13:54:41.148309 9285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:41 dist-test-slave-q25r krb5kdc[9272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:41.152750 9285 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:41.154960 9285 webserver.cc:460] Webserver started at http://127.0.0.1:44811/ using document root <none> and password file <none>
I20260927 13:54:41.155905 9285 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:41.156051 9285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:41.156459 9285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:41.160387 9285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "39a1dd8d20ad43b2b98140987022b33d"
format_stamp: "Formatted at 2026-09-27 13:54:41 on dist-test-slave-q25r"
server_key: "33c5fe01d4ca8c37fc367962163e5783"
server_key_iv: "8d0aacf4e0e7d977c0d5166245f73499"
server_key_version: "encryptionkey@0"
I20260927 13:54:41.161351 9285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "39a1dd8d20ad43b2b98140987022b33d"
format_stamp: "Formatted at 2026-09-27 13:54:41 on dist-test-slave-q25r"
server_key: "33c5fe01d4ca8c37fc367962163e5783"
server_key_iv: "8d0aacf4e0e7d977c0d5166245f73499"
server_key_version: "encryptionkey@0"
I20260927 13:54:41.168206 9285 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 13:54:41.172709 9300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:41.174606 9285 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:54:41.174844 9285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "39a1dd8d20ad43b2b98140987022b33d"
format_stamp: "Formatted at 2026-09-27 13:54:41 on dist-test-slave-q25r"
server_key: "33c5fe01d4ca8c37fc367962163e5783"
server_key_iv: "8d0aacf4e0e7d977c0d5166245f73499"
server_key_version: "encryptionkey@0"
I20260927 13:54:41.175084 9285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:41.210870 9285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:41.216449 9285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:41.216800 9285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:41.239621 9285 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45989
I20260927 13:54:41.239718 9351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45989 every 8 connection(s)
I20260927 13:54:41.241444 9285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:41.244536 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 9285
I20260927 13:54:41.244728 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:41.245237 21013 external_mini_cluster.cc:1428] Setting key 19efd42bfee0a61dd61c53483c147da9
I20260927 13:54:41.245677 9352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:41.261605 9352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d: Bootstrap starting.
Sep 27 13:54:41 dist-test-slave-q25r krb5kdc[9272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 13:54:41.265223 9352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:41.266530 9352 log.cc:824] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:41.269738 9352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d: No bootstrap required, opened a new log
I20260927 13:54:41.273837 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:41.247460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44351 (local address 127.0.0.1:45989)
0927 13:54:41.248485 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:41.248499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:41.248557 (+ 58us) server_negotiation.cc:407] Connection header received
0927 13:54:41.250774 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:41.250805 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:41.251386 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:41.251851 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:41.254208 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:41.255746 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:41.257747 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:41.258632 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:41.258829 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:41.264462 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:41.264476 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:41.264499 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:41.264564 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:41.268216 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:41.269276 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:41.269286 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:41.269294 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:41.269518 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:41.270053 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:41.270061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:41.270065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:41.270945 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:41.271315 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:41.271791 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 13:54:41.272327 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":431,"threads_started":1}
I20260927 13:54:41.279325 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:41.279369 9352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1dd8d20ad43b2b98140987022b33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260927 13:54:41.279897 9352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:41.280007 9352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a1dd8d20ad43b2b98140987022b33d, State: Initialized, Role: FOLLOWER
I20260927 13:54:41.280689 9352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1dd8d20ad43b2b98140987022b33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260927 13:54:41.280920 9352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:41.281054 9352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:41.281224 9352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:41.283660 9352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1dd8d20ad43b2b98140987022b33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260927 13:54:41.284225 9352 leader_election.cc:302] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39a1dd8d20ad43b2b98140987022b33d; no voters:
I20260927 13:54:41.284981 9352 leader_election.cc:288] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:41.285163 9361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:41.286628 9361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [term 1 LEADER]: Becoming Leader. State: Replica: 39a1dd8d20ad43b2b98140987022b33d, State: Running, Role: LEADER
I20260927 13:54:41.287313 9361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1dd8d20ad43b2b98140987022b33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260927 13:54:41.287863 9352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:41.290377 9363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39a1dd8d20ad43b2b98140987022b33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1dd8d20ad43b2b98140987022b33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } } }
I20260927 13:54:41.290848 9363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:41.291443 9364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39a1dd8d20ad43b2b98140987022b33d. Latest consensus state: current_term: 1 leader_uuid: "39a1dd8d20ad43b2b98140987022b33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1dd8d20ad43b2b98140987022b33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } } }
I20260927 13:54:41.291738 9364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:41.293944 9367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:41.300836 9367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:41.304977 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:41.284760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52091 (local address 127.0.0.1:45989)
0927 13:54:41.285013 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:41.285022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:41.285568 (+ 546us) server_negotiation.cc:407] Connection header received
0927 13:54:41.285764 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:41.285770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:41.285880 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:41.286007 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:41.287914 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:41.288842 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:41.290670 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:41.291408 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:41.291522 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:41.295223 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:41.295234 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:41.295241 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:41.295300 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:41.299851 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:41.301241 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:41.301247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:41.301251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:41.301434 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:41.301947 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:41.301953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:41.301957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:41.302458 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:41.304032 (+ 1574us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:41.304215 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 13:54:41.304515 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 13:54:41.315721 9367 catalog_manager.cc:1347] Generated new cluster ID: 3a799f43657540bb9168d21f4069e81c
I20260927 13:54:41.315897 9367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:41.336855 9367 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:41.338517 9367 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:41.352905 9367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39a1dd8d20ad43b2b98140987022b33d: Generated new TSK 0
I20260927 13:54:41.354499 9367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:41.364537 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9285
I20260927 13:54:41.366341 9347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:41.461346 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 9285
2026-09-27T13:54:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2067 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:41 dist-test-slave-q25r krb5kdc[9384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:41 dist-test-slave-q25r krb5kdc[9384](info): set up 2 sockets
Sep 27 13:54:41 dist-test-slave-q25r krb5kdc[9384](info): commencing operation
krb5kdc: starting...
W20260927 13:54:42.513046 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:42 dist-test-slave-q25r krb5kdc[9384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:42.824154 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:43.225523 9397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:43.225936 9397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:43.226079 9397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:43.235222 9397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:43.235414 9397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:43.235503 9397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:43.235594 9397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:43.235662 9397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:43.247226 9397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:43.249032 9397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:43.250610 9397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:43.264595 9405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:43.264698 9402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:43.264676 9403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:43.266021 9397 server_base.cc:1034] running on GCE node
I20260927 13:54:43.267035 9397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:43.268966 9397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:43.270218 9397 hybrid_clock.cc:648] HybridClock initialized: now 1790517283270163 us; error 46 us; skew 500 ppm
I20260927 13:54:43.270732 9397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:43 dist-test-slave-q25r krb5kdc[9384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:43.274799 9397 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:43.276824 9397 webserver.cc:460] Webserver started at http://127.0.0.1:43939/ using document root <none> and password file <none>
I20260927 13:54:43.277693 9397 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:43.277819 9397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:43.278198 9397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:43.281977 9397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "d3c7afa0acc3492591b9e43186405527"
format_stamp: "Formatted at 2026-09-27 13:54:43 on dist-test-slave-q25r"
server_key: "c78eaed5a14400232db896fd1bb3c8ca"
server_key_iv: "8743f17328104415ca7e2e52fcdcce3d"
server_key_version: "encryptionkey@0"
I20260927 13:54:43.282883 9397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "d3c7afa0acc3492591b9e43186405527"
format_stamp: "Formatted at 2026-09-27 13:54:43 on dist-test-slave-q25r"
server_key: "c78eaed5a14400232db896fd1bb3c8ca"
server_key_iv: "8743f17328104415ca7e2e52fcdcce3d"
server_key_version: "encryptionkey@0"
I20260927 13:54:43.289867 9397 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260927 13:54:43.294454 9412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:43.296171 9397 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 13:54:43.296484 9397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "d3c7afa0acc3492591b9e43186405527"
format_stamp: "Formatted at 2026-09-27 13:54:43 on dist-test-slave-q25r"
server_key: "c78eaed5a14400232db896fd1bb3c8ca"
server_key_iv: "8743f17328104415ca7e2e52fcdcce3d"
server_key_version: "encryptionkey@0"
I20260927 13:54:43.296726 9397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:43.312072 9397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:43.317937 9397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:43.318284 9397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:43.340183 9397 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39317
I20260927 13:54:43.340253 9463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39317 every 8 connection(s)
I20260927 13:54:43.341918 9397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:43.345645 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 9397
I20260927 13:54:43.345829 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:43.346377 21013 external_mini_cluster.cc:1428] Setting key eda484ff8b6e2a090792bcd73199e2e0
I20260927 13:54:43.346457 9464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:43.363559 9464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527: Bootstrap starting.
I20260927 13:54:43.364068 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:43.348754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57889 (local address 127.0.0.1:39317)
0927 13:54:43.349898 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:43.349918 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:43.350006 (+ 88us) server_negotiation.cc:407] Connection header received
0927 13:54:43.352122 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:43.352159 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:43.352828 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:43.353410 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:43.354110 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:43.356260 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:43.357932 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:43.359117 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:43.359390 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:43.360758 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:43.360771 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:43.360795 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:43.361690 (+ 895us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:43.361807 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:43.361848 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:43.362046 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:43.362263 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 13:54:43.362588 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":505,"threads_started":1}
I20260927 13:54:43.367540 9464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:43.369051 9464 log.cc:824] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:43.369257 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:43.372911 9464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527: No bootstrap required, opened a new log
I20260927 13:54:43.382337 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:43.374628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33929 (local address 127.0.0.1:39317)
0927 13:54:43.374934 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:43.374961 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:43.375471 (+ 510us) server_negotiation.cc:407] Connection header received
0927 13:54:43.375647 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:43.375653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:43.375758 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:43.375934 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:43.376607 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:43.377797 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:43.379281 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:43.380004 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:43.380160 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:43.381373 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:43.381381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:43.381387 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:43.381420 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:43.381478 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:43.381499 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:43.381618 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:43.381813 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 13:54:43.381936 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:54:43.383508 9464 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c7afa0acc3492591b9e43186405527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } }
I20260927 13:54:43.384133 9464 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:43.384261 9464 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c7afa0acc3492591b9e43186405527, State: Initialized, Role: FOLLOWER
I20260927 13:54:43.385006 9464 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c7afa0acc3492591b9e43186405527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } }
I20260927 13:54:43.385291 9464 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:43.385440 9464 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:43.385618 9464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:43.388439 9464 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c7afa0acc3492591b9e43186405527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } }
I20260927 13:54:43.389140 9464 leader_election.cc:302] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c7afa0acc3492591b9e43186405527; no voters:
I20260927 13:54:43.389911 9464 leader_election.cc:288] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:43.390130 9474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:43.391367 9474 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [term 1 LEADER]: Becoming Leader. State: Replica: d3c7afa0acc3492591b9e43186405527, State: Running, Role: LEADER
I20260927 13:54:43.392040 9474 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c7afa0acc3492591b9e43186405527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } }
I20260927 13:54:43.392722 9464 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:43.395483 9475 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c7afa0acc3492591b9e43186405527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c7afa0acc3492591b9e43186405527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } } }
I20260927 13:54:43.395911 9475 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:43.397668 9482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:43.397969 9476 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c7afa0acc3492591b9e43186405527. Latest consensus state: current_term: 1 leader_uuid: "d3c7afa0acc3492591b9e43186405527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c7afa0acc3492591b9e43186405527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } } }
I20260927 13:54:43.398388 9476 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:43.405496 9482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:43.419276 9482 catalog_manager.cc:1347] Generated new cluster ID: cbae691a8e45430283ee9f9b07833968
I20260927 13:54:43.419430 9482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:43.436898 9482 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:43.438867 9482 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:43.458428 9482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c7afa0acc3492591b9e43186405527: Generated new TSK 0
I20260927 13:54:43.459668 9482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:43.470651 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9397
W20260927 13:54:43.472762 9459 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57889
W20260927 13:54:43.473742 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9397: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:54:43.473981 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 9397
2026-09-27T13:54:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:43 dist-test-slave-q25r krb5kdc[9495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:43 dist-test-slave-q25r krb5kdc[9495](info): set up 2 sockets
Sep 27 13:54:43 dist-test-slave-q25r krb5kdc[9495](info): commencing operation
krb5kdc: starting...
W20260927 13:54:44.525738 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:44 dist-test-slave-q25r krb5kdc[9495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:44.828769 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:45.198347 9508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:45.198671 9508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:45.198770 9508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:45.206377 9508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:45.206519 9508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:45.206570 9508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:45.206645 9508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:45.206696 9508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:45.217139 9508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:45.218480 9508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:45.219837 9508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:45.232939 9514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:45.233063 9516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:45.233043 9513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:45.234905 9508 server_base.cc:1034] running on GCE node
I20260927 13:54:45.236038 9508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:45.238168 9508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:45.239441 9508 hybrid_clock.cc:648] HybridClock initialized: now 1790517285239302 us; error 117 us; skew 500 ppm
I20260927 13:54:45.240007 9508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:45 dist-test-slave-q25r krb5kdc[9495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:45.248531 9508 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:45.251225 9508 webserver.cc:460] Webserver started at http://127.0.0.1:38065/ using document root <none> and password file <none>
I20260927 13:54:45.252447 9508 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:45.252625 9508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:45.253005 9508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:45.259243 9508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "eaf74b85eef34bd1935a5e96870f5af7"
format_stamp: "Formatted at 2026-09-27 13:54:45 on dist-test-slave-q25r"
server_key: "8c9e382d640f6cac4ef38096eca6ce90"
server_key_iv: "005f5e6191c4e45284ca420a56990886"
server_key_version: "encryptionkey@0"
I20260927 13:54:45.260598 9508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "eaf74b85eef34bd1935a5e96870f5af7"
format_stamp: "Formatted at 2026-09-27 13:54:45 on dist-test-slave-q25r"
server_key: "8c9e382d640f6cac4ef38096eca6ce90"
server_key_iv: "005f5e6191c4e45284ca420a56990886"
server_key_version: "encryptionkey@0"
I20260927 13:54:45.268987 9508 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260927 13:54:45.274923 9523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:45.277192 9508 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 13:54:45.277403 9508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "eaf74b85eef34bd1935a5e96870f5af7"
format_stamp: "Formatted at 2026-09-27 13:54:45 on dist-test-slave-q25r"
server_key: "8c9e382d640f6cac4ef38096eca6ce90"
server_key_iv: "005f5e6191c4e45284ca420a56990886"
server_key_version: "encryptionkey@0"
I20260927 13:54:45.277662 9508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:45.285524 9508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:45.291712 9508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:45.292070 9508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:45.313781 9508 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43029
I20260927 13:54:45.313831 9574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43029 every 8 connection(s)
I20260927 13:54:45.315565 9508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:45.319829 9575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:45.321153 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 9508
I20260927 13:54:45.321359 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:45.321864 21013 external_mini_cluster.cc:1428] Setting key a6b412074e25468664d9aabcc68ce4ba
I20260927 13:54:45.331408 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:45.324301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44765 (local address 127.0.0.1:43029)
0927 13:54:45.325368 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:45.325395 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:45.325468 (+ 73us) server_negotiation.cc:407] Connection header received
0927 13:54:45.327193 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:45.327224 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:45.327762 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:45.328261 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:45.329373 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:45.329381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:45.329397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:45.329417 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:45.329493 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:45.329527 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:45.329791 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:45.329927 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 13:54:45.330147 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":386,"threads_started":1}
I20260927 13:54:45.335165 9575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7: Bootstrap starting.
I20260927 13:54:45.336486 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:45.338565 9575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:45.339677 9575 log.cc:824] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:45.342916 9575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7: No bootstrap required, opened a new log
I20260927 13:54:45.344187 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:45.341532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54703 (local address 127.0.0.1:43029)
0927 13:54:45.341723 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:45.341729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:45.342066 (+ 337us) server_negotiation.cc:407] Connection header received
0927 13:54:45.342224 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:45.342230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:45.342294 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:45.342442 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:45.343467 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:45.343473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:45.343478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:45.343502 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:45.343556 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:45.343577 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:45.343721 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:45.343838 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 13:54:45.343903 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 13:54:45.351285 9575 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf74b85eef34bd1935a5e96870f5af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43029 } }
I20260927 13:54:45.351706 9575 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:45.351786 9575 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf74b85eef34bd1935a5e96870f5af7, State: Initialized, Role: FOLLOWER
I20260927 13:54:45.352438 9575 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf74b85eef34bd1935a5e96870f5af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43029 } }
I20260927 13:54:45.352685 9575 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:45.352793 9575 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:45.352928 9575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:45.355454 9575 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf74b85eef34bd1935a5e96870f5af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43029 } }
I20260927 13:54:45.355993 9575 leader_election.cc:302] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaf74b85eef34bd1935a5e96870f5af7; no voters:
I20260927 13:54:45.356655 9575 leader_election.cc:288] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:45.356877 9585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:45.357831 9585 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [term 1 LEADER]: Becoming Leader. State: Replica: eaf74b85eef34bd1935a5e96870f5af7, State: Running, Role: LEADER
I20260927 13:54:45.358594 9585 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf74b85eef34bd1935a5e96870f5af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43029 } }
I20260927 13:54:45.359151 9575 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:45.362295 9586 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaf74b85eef34bd1935a5e96870f5af7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf74b85eef34bd1935a5e96870f5af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43029 } } }
I20260927 13:54:45.362735 9586 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:45.363708 9592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:45.363510 9587 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaf74b85eef34bd1935a5e96870f5af7. Latest consensus state: current_term: 1 leader_uuid: "eaf74b85eef34bd1935a5e96870f5af7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf74b85eef34bd1935a5e96870f5af7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43029 } } }
I20260927 13:54:45.364570 9587 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:45.371253 9592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:45.384042 9592 catalog_manager.cc:1347] Generated new cluster ID: 3dfa64c48c5c498ba0ab07f8ee4197c4
I20260927 13:54:45.384178 9592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:45.394778 9592 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:45.395404 9592 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:45.407958 9592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaf74b85eef34bd1935a5e96870f5af7: Generated new TSK 0
I20260927 13:54:45.409608 9592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:45.422079 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9508
W20260927 13:54:45.423751 9570 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44765
W20260927 13:54:45.424937 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9508: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:54:45.425184 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 9508
2026-09-27T13:54:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:45 dist-test-slave-q25r krb5kdc[9606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:45 dist-test-slave-q25r krb5kdc[9606](info): set up 2 sockets
Sep 27 13:54:45 dist-test-slave-q25r krb5kdc[9606](info): commencing operation
krb5kdc: starting...
W20260927 13:54:46.476940 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:46 dist-test-slave-q25r krb5kdc[9606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:46.773986 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:47.125032 9619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:47.125378 9619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:47.125520 9619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:47.133119 9619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:47.133282 9619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:47.133368 9619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:47.133455 9619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:47.133519 9619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:47.143777 9619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:47.145187 9619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:47.146479 9619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:47.159188 9625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:47.159096 9624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:47.159242 9627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:47.160614 9619 server_base.cc:1034] running on GCE node
I20260927 13:54:47.161630 9619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:47.163295 9619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:47.164541 9619 hybrid_clock.cc:648] HybridClock initialized: now 1790517287164484 us; error 43 us; skew 500 ppm
I20260927 13:54:47.164947 9619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:47 dist-test-slave-q25r krb5kdc[9606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:47.168675 9619 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:47.170770 9619 webserver.cc:460] Webserver started at http://127.0.0.1:36473/ using document root <none> and password file <none>
I20260927 13:54:47.171702 9619 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:47.171844 9619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:47.172217 9619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:47.176064 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "4126c55f4a9c4a079b9dcb82d89ab6ec"
format_stamp: "Formatted at 2026-09-27 13:54:47 on dist-test-slave-q25r"
server_key: "23a5a1fae13fef5fde372e345dadbc03"
server_key_iv: "c404ffc3c73810a560c7a0a37f958fc2"
server_key_version: "encryptionkey@0"
I20260927 13:54:47.177076 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "4126c55f4a9c4a079b9dcb82d89ab6ec"
format_stamp: "Formatted at 2026-09-27 13:54:47 on dist-test-slave-q25r"
server_key: "23a5a1fae13fef5fde372e345dadbc03"
server_key_iv: "c404ffc3c73810a560c7a0a37f958fc2"
server_key_version: "encryptionkey@0"
I20260927 13:54:47.184616 9619 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 13:54:47.189658 9634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:47.191715 9619 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:54:47.191998 9619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "4126c55f4a9c4a079b9dcb82d89ab6ec"
format_stamp: "Formatted at 2026-09-27 13:54:47 on dist-test-slave-q25r"
server_key: "23a5a1fae13fef5fde372e345dadbc03"
server_key_iv: "c404ffc3c73810a560c7a0a37f958fc2"
server_key_version: "encryptionkey@0"
I20260927 13:54:47.192260 9619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:47.221180 9619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:47.226531 9619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:47.226859 9619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:47.249608 9619 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36921
I20260927 13:54:47.249684 9685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36921 every 8 connection(s)
I20260927 13:54:47.251487 9619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:47.255908 9686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:47.256266 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 9619
I20260927 13:54:47.256487 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:47.257045 21013 external_mini_cluster.cc:1428] Setting key 098f8bd0cb15c575f41d041e77879629
I20260927 13:54:47.271166 9686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec: Bootstrap starting.
Sep 27 13:54:47 dist-test-slave-q25r krb5kdc[9606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 13:54:47.275244 9686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:47.276597 9686 log.cc:824] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:47.280804 9686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec: No bootstrap required, opened a new log
I20260927 13:54:47.285785 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:47.259354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:41673 (local address 127.0.0.1:36921)
0927 13:54:47.260329 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:47.260384 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:47.260452 (+ 68us) server_negotiation.cc:407] Connection header received
0927 13:54:47.262707 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:47.262744 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:47.263404 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:47.263856 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:47.265906 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:47.267523 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:47.269459 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:47.270299 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:47.270480 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:47.275935 (+ 5455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:47.275948 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:47.275969 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:47.276040 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:47.280129 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:47.281417 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:47.281427 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:47.281435 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:47.281683 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:47.282219 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:47.282225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:47.282228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:47.282986 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:47.283370 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:47.283899 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 13:54:47.284363 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":407,"threads_started":1}
I20260927 13:54:47.290901 9686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4126c55f4a9c4a079b9dcb82d89ab6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 13:54:47.291204 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:47.291338 9686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:47.291448 9686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4126c55f4a9c4a079b9dcb82d89ab6ec, State: Initialized, Role: FOLLOWER
I20260927 13:54:47.292109 9686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4126c55f4a9c4a079b9dcb82d89ab6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 13:54:47.292443 9686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:47.292590 9686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:47.292759 9686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:47.295953 9686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4126c55f4a9c4a079b9dcb82d89ab6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 13:54:47.296650 9686 leader_election.cc:302] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4126c55f4a9c4a079b9dcb82d89ab6ec; no voters:
I20260927 13:54:47.297326 9686 leader_election.cc:288] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:47.297441 9695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:47.298110 9695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [term 1 LEADER]: Becoming Leader. State: Replica: 4126c55f4a9c4a079b9dcb82d89ab6ec, State: Running, Role: LEADER
I20260927 13:54:47.298823 9695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4126c55f4a9c4a079b9dcb82d89ab6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 13:54:47.302531 9686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:47.304149 9697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4126c55f4a9c4a079b9dcb82d89ab6ec. Latest consensus state: current_term: 1 leader_uuid: "4126c55f4a9c4a079b9dcb82d89ab6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4126c55f4a9c4a079b9dcb82d89ab6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } } }
I20260927 13:54:47.304628 9697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:47.305312 9698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4126c55f4a9c4a079b9dcb82d89ab6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4126c55f4a9c4a079b9dcb82d89ab6ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } } }
I20260927 13:54:47.305624 9698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:47.308473 9701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:47.317171 9701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:47.317492 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:47.297091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:51709 (local address 127.0.0.1:36921)
0927 13:54:47.297338 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:47.297345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:47.297968 (+ 623us) server_negotiation.cc:407] Connection header received
0927 13:54:47.298476 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:47.298483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:47.298583 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:47.298661 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:47.300909 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:47.301809 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:47.303429 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:47.304141 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:47.304249 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:47.307981 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:47.307989 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:47.307995 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:47.308042 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:47.312493 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:47.314039 (+ 1546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:47.314046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:47.314051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:47.314279 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:47.314955 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:47.314961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:47.314964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:47.315500 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:47.315759 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:47.316648 (+ 889us) server_negotiation.cc:299] Negotiation successful
0927 13:54:47.316918 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 13:54:47.336050 9701 catalog_manager.cc:1347] Generated new cluster ID: 13f9d5030c5f4468856ee18a36f1db6e
I20260927 13:54:47.336225 9701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:47.353979 9701 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:47.355392 9701 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:47.365203 9701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4126c55f4a9c4a079b9dcb82d89ab6ec: Generated new TSK 0
I20260927 13:54:47.366125 9701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:47.378159 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9619
I20260927 13:54:47.379760 9681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:47.483158 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 9619
W20260927 13:54:47.501305 9604 connection.cc:535] client connection to 127.0.0.1:36921 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36921): Connection reset by peer (error 104)
2026-09-27T13:54:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2060 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:47 dist-test-slave-q25r krb5kdc[9718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:47 dist-test-slave-q25r krb5kdc[9718](info): set up 2 sockets
Sep 27 13:54:47 dist-test-slave-q25r krb5kdc[9718](info): commencing operation
krb5kdc: starting...
W20260927 13:54:48.532565 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:48 dist-test-slave-q25r krb5kdc[9718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:48.835817 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:49.181597 9731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:49.181917 9731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:49.182037 9731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:49.189541 9731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:49.189682 9731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:49.189738 9731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:49.189831 9731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:49.189898 9731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:49.200060 9731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:49.201422 9731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:49.202708 9731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:49.215591 9739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:49.215869 9736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:49.215938 9737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:49.216449 9731 server_base.cc:1034] running on GCE node
I20260927 13:54:49.217555 9731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:49.219087 9731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:49.220323 9731 hybrid_clock.cc:648] HybridClock initialized: now 1790517289220264 us; error 58 us; skew 500 ppm
I20260927 13:54:49.220721 9731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:49 dist-test-slave-q25r krb5kdc[9718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:49.223917 9731 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:49.225653 9731 webserver.cc:460] Webserver started at http://127.0.0.1:45959/ using document root <none> and password file <none>
I20260927 13:54:49.226372 9731 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:49.226488 9731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:49.226800 9731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:49.229907 9731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "7633404bbc1846f29b80bbfe367cc259"
format_stamp: "Formatted at 2026-09-27 13:54:49 on dist-test-slave-q25r"
server_key: "548a355abe25c423ad228a142eb07a69"
server_key_iv: "3fb4152212d5b462af3281fab706f499"
server_key_version: "encryptionkey@0"
I20260927 13:54:49.230703 9731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "7633404bbc1846f29b80bbfe367cc259"
format_stamp: "Formatted at 2026-09-27 13:54:49 on dist-test-slave-q25r"
server_key: "548a355abe25c423ad228a142eb07a69"
server_key_iv: "3fb4152212d5b462af3281fab706f499"
server_key_version: "encryptionkey@0"
I20260927 13:54:49.236632 9731 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 13:54:49.240703 9746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:49.242601 9731 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:54:49.242831 9731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "7633404bbc1846f29b80bbfe367cc259"
format_stamp: "Formatted at 2026-09-27 13:54:49 on dist-test-slave-q25r"
server_key: "548a355abe25c423ad228a142eb07a69"
server_key_iv: "3fb4152212d5b462af3281fab706f499"
server_key_version: "encryptionkey@0"
I20260927 13:54:49.243077 9731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:49.259143 9731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:49.264015 9731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:49.264312 9731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:49.284556 9731 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42575
I20260927 13:54:49.284616 9797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42575 every 8 connection(s)
I20260927 13:54:49.286051 9731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:49.287433 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 9731
I20260927 13:54:49.287603 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:49.288055 21013 external_mini_cluster.cc:1428] Setting key 7ea01f70940fee098708a03e049a5043
I20260927 13:54:49.291389 9798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:49.303288 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:49.290230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:51449 (local address 127.0.0.1:42575)
0927 13:54:49.291057 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:49.291077 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:49.291122 (+ 45us) server_negotiation.cc:407] Connection header received
0927 13:54:49.292951 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:49.292977 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:49.293429 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:49.293794 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:49.294637 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:49.296069 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:49.298159 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:49.298948 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:49.299126 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:49.300714 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:49.300724 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:49.300741 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:49.301357 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:49.301459 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:49.301487 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:49.301645 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:49.301992 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 13:54:49.302165 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":371,"threads_started":1}
I20260927 13:54:49.306205 9798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259: Bootstrap starting.
I20260927 13:54:49.308209 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:49.310154 9798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:49.311539 9798 log.cc:824] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:49.315744 9798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259: No bootstrap required, opened a new log
I20260927 13:54:49.321033 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:49.312877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:49643 (local address 127.0.0.1:42575)
0927 13:54:49.313121 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:49.313130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:49.313743 (+ 613us) server_negotiation.cc:407] Connection header received
0927 13:54:49.313977 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:49.313981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:49.314061 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:49.314155 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:49.314873 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:49.315714 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:49.317802 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:49.318419 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:49.318576 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:49.320043 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:49.320048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:49.320052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:49.320074 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:49.320150 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:49.320170 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:49.320300 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:49.320562 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 13:54:49.320623 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 13:54:49.326319 9798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633404bbc1846f29b80bbfe367cc259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42575 } }
I20260927 13:54:49.326881 9798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:49.327044 9798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7633404bbc1846f29b80bbfe367cc259, State: Initialized, Role: FOLLOWER
I20260927 13:54:49.327795 9798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633404bbc1846f29b80bbfe367cc259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42575 } }
I20260927 13:54:49.328054 9798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:49.328218 9798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:49.328490 9798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:49.331419 9798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633404bbc1846f29b80bbfe367cc259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42575 } }
I20260927 13:54:49.332016 9798 leader_election.cc:302] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7633404bbc1846f29b80bbfe367cc259; no voters:
I20260927 13:54:49.332739 9798 leader_election.cc:288] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:49.332929 9808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:49.333724 9808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [term 1 LEADER]: Becoming Leader. State: Replica: 7633404bbc1846f29b80bbfe367cc259, State: Running, Role: LEADER
I20260927 13:54:49.334340 9808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633404bbc1846f29b80bbfe367cc259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42575 } }
I20260927 13:54:49.335206 9798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:49.339839 9809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7633404bbc1846f29b80bbfe367cc259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633404bbc1846f29b80bbfe367cc259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42575 } } }
I20260927 13:54:49.339254 9810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7633404bbc1846f29b80bbfe367cc259. Latest consensus state: current_term: 1 leader_uuid: "7633404bbc1846f29b80bbfe367cc259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7633404bbc1846f29b80bbfe367cc259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42575 } } }
I20260927 13:54:49.340914 9810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:49.340914 9809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:49.342105 9820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:49.347767 9820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:49.360285 9820 catalog_manager.cc:1347] Generated new cluster ID: 690ffa73b4c84b4aac66a76c66235890
I20260927 13:54:49.360639 9820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:49.373991 9820 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:49.375716 9820 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:49.388764 9820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7633404bbc1846f29b80bbfe367cc259: Generated new TSK 0
I20260927 13:54:49.390026 9820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:49.403599 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9731
W20260927 13:54:49.405139 9793 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.34:51449
W20260927 13:54:49.406121 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9731: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:54:49.406273 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 9731
W20260927 13:54:49.424002 9716 connection.cc:535] client connection to 127.0.0.1:42575 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42575): Connection reset by peer (error 104)
2026-09-27T13:54:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:49 dist-test-slave-q25r krb5kdc[9829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:49 dist-test-slave-q25r krb5kdc[9829](info): set up 2 sockets
Sep 27 13:54:49 dist-test-slave-q25r krb5kdc[9829](info): commencing operation
krb5kdc: starting...
W20260927 13:54:50.461045 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.006s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:50 dist-test-slave-q25r krb5kdc[9829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:50.763270 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:51.123450 9842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:51.123813 9842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:51.123952 9842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:51.131486 9842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:51.131646 9842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:51.131727 9842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:51.131814 9842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:51.131877 9842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:51.142055 9842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:51.143467 9842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:51.145117 9842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:51.158931 9847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:51.158911 9848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:51.159476 9850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:51.159914 9842 server_base.cc:1034] running on GCE node
I20260927 13:54:51.160887 9842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:51.162531 9842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:51.163772 9842 hybrid_clock.cc:648] HybridClock initialized: now 1790517291163713 us; error 60 us; skew 500 ppm
I20260927 13:54:51.164177 9842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:51 dist-test-slave-q25r krb5kdc[9829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:51.171378 9842 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:51.173252 9842 webserver.cc:460] Webserver started at http://127.0.0.1:45207/ using document root <none> and password file <none>
I20260927 13:54:51.173974 9842 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:51.174099 9842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:51.174393 9842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:51.177517 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "c427e6d117004c9db31c49194a93f819"
format_stamp: "Formatted at 2026-09-27 13:54:51 on dist-test-slave-q25r"
server_key: "7bfecf1e6775cdf0fd7a868e83578e91"
server_key_iv: "57c6f3363f2b7f475d82c3939ced54b3"
server_key_version: "encryptionkey@0"
I20260927 13:54:51.178295 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "c427e6d117004c9db31c49194a93f819"
format_stamp: "Formatted at 2026-09-27 13:54:51 on dist-test-slave-q25r"
server_key: "7bfecf1e6775cdf0fd7a868e83578e91"
server_key_iv: "57c6f3363f2b7f475d82c3939ced54b3"
server_key_version: "encryptionkey@0"
I20260927 13:54:51.184094 9842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:54:51.188069 9857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:51.189752 9842 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:54:51.189949 9842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "c427e6d117004c9db31c49194a93f819"
format_stamp: "Formatted at 2026-09-27 13:54:51 on dist-test-slave-q25r"
server_key: "7bfecf1e6775cdf0fd7a868e83578e91"
server_key_iv: "57c6f3363f2b7f475d82c3939ced54b3"
server_key_version: "encryptionkey@0"
I20260927 13:54:51.190155 9842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:51.196205 9842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:51.200724 9842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:51.201057 9842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:51.221635 9842 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32849
I20260927 13:54:51.221719 9908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32849 every 8 connection(s)
I20260927 13:54:51.223073 9842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:51.224599 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 9842
I20260927 13:54:51.224766 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:51.225198 21013 external_mini_cluster.cc:1428] Setting key 51d4e5344d5fe7dad750aca4a97da4bb
I20260927 13:54:51.228744 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:51.235908 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:51.227292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:60337 (local address 127.0.0.1:32849)
0927 13:54:51.228256 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:51.228276 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:51.228365 (+ 89us) server_negotiation.cc:407] Connection header received
0927 13:54:51.230935 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:51.230972 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:51.231547 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:51.232111 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:51.233413 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:51.233422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:51.233441 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:51.233468 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:51.233569 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:51.233601 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:51.233889 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:51.234049 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 13:54:51.234316 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":365,"threads_started":1}
I20260927 13:54:51.240603 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:51.244580 9909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819: Bootstrap starting.
I20260927 13:54:51.248160 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:51.245455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:43697 (local address 127.0.0.1:32849)
0927 13:54:51.245657 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:51.245664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:51.246078 (+ 414us) server_negotiation.cc:407] Connection header received
0927 13:54:51.246267 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:51.246272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:51.246330 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:51.246414 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:51.247492 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:51.247496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:51.247499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:51.247514 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:51.247562 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:51.247579 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:51.247684 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:51.247783 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 13:54:51.247854 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:54:51.249339 9909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:51.250715 9909 log.cc:824] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:51.254271 9909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819: No bootstrap required, opened a new log
I20260927 13:54:51.263525 9909 raft_consensus.cc:348] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c427e6d117004c9db31c49194a93f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260927 13:54:51.264178 9909 raft_consensus.cc:374] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:51.264307 9909 raft_consensus.cc:729] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c427e6d117004c9db31c49194a93f819, State: Initialized, Role: FOLLOWER
I20260927 13:54:51.265224 9909 consensus_queue.cc:260] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c427e6d117004c9db31c49194a93f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260927 13:54:51.265550 9909 raft_consensus.cc:388] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:51.265728 9909 raft_consensus.cc:482] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:51.265929 9909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:51.268829 9909 raft_consensus.cc:504] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c427e6d117004c9db31c49194a93f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260927 13:54:51.269454 9909 leader_election.cc:302] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c427e6d117004c9db31c49194a93f819; no voters:
I20260927 13:54:51.270107 9909 leader_election.cc:288] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:51.270318 9919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:51.271096 9919 raft_consensus.cc:686] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [term 1 LEADER]: Becoming Leader. State: Replica: c427e6d117004c9db31c49194a93f819, State: Running, Role: LEADER
I20260927 13:54:51.271840 9919 consensus_queue.cc:237] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c427e6d117004c9db31c49194a93f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260927 13:54:51.272966 9909 sys_catalog.cc:564] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:51.275139 9921 sys_catalog.cc:455] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c427e6d117004c9db31c49194a93f819. Latest consensus state: current_term: 1 leader_uuid: "c427e6d117004c9db31c49194a93f819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c427e6d117004c9db31c49194a93f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } } }
I20260927 13:54:51.275928 9921 sys_catalog.cc:458] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:51.275204 9920 sys_catalog.cc:455] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c427e6d117004c9db31c49194a93f819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c427e6d117004c9db31c49194a93f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } } }
I20260927 13:54:51.276610 9920 sys_catalog.cc:458] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:51.277838 9926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:51.288938 9926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:51.301427 9926 catalog_manager.cc:1347] Generated new cluster ID: 6f6c0102feda4f77bdf0dcf67e96d26a
I20260927 13:54:51.301580 9926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:51.324019 9926 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:51.324600 9926 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:51.336838 9926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c427e6d117004c9db31c49194a93f819: Generated new TSK 0
I20260927 13:54:51.337787 9926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:51.342831 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9842
W20260927 13:54:51.344715 9904 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.34:60337
W20260927 13:54:51.345897 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9842: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:54:51.346150 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 9842
2026-09-27T13:54:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:51 dist-test-slave-q25r krb5kdc[9940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:51 dist-test-slave-q25r krb5kdc[9940](info): set up 2 sockets
Sep 27 13:54:51 dist-test-slave-q25r krb5kdc[9940](info): commencing operation
krb5kdc: starting...
W20260927 13:54:52.405295 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:52 dist-test-slave-q25r krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:52.702039 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:53.040840 9953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:53.041174 9953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:53.041270 9953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:53.049970 9953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:53.050133 9953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:53.050185 9953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:53.050283 9953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:53.050340 9953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:53.061991 9953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:53.063452 9953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:53.064839 9953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:53.077095 9959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:53.077224 9958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:53.077811 9961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:53.078704 9953 server_base.cc:1034] running on GCE node
I20260927 13:54:53.079653 9953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:53.081290 9953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:53.082541 9953 hybrid_clock.cc:648] HybridClock initialized: now 1790517293082477 us; error 55 us; skew 500 ppm
I20260927 13:54:53.082940 9953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:53 dist-test-slave-q25r krb5kdc[9940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:53.086545 9953 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:53.089062 9953 webserver.cc:460] Webserver started at http://127.0.0.1:39201/ using document root <none> and password file <none>
I20260927 13:54:53.089836 9953 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:53.089965 9953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:53.090271 9953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:53.093514 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "83d47d07d7bc4f4991afc46e469ef282"
format_stamp: "Formatted at 2026-09-27 13:54:53 on dist-test-slave-q25r"
server_key: "4bcd4dd9d15de924d756d64f48fa6c88"
server_key_iv: "5169bff604fb0d7297eb0d4d6d2a4b2c"
server_key_version: "encryptionkey@0"
I20260927 13:54:53.094295 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "83d47d07d7bc4f4991afc46e469ef282"
format_stamp: "Formatted at 2026-09-27 13:54:53 on dist-test-slave-q25r"
server_key: "4bcd4dd9d15de924d756d64f48fa6c88"
server_key_iv: "5169bff604fb0d7297eb0d4d6d2a4b2c"
server_key_version: "encryptionkey@0"
I20260927 13:54:53.100082 9953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:54:53.104127 9968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:53.105827 9953 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:54:53.106053 9953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "83d47d07d7bc4f4991afc46e469ef282"
format_stamp: "Formatted at 2026-09-27 13:54:53 on dist-test-slave-q25r"
server_key: "4bcd4dd9d15de924d756d64f48fa6c88"
server_key_iv: "5169bff604fb0d7297eb0d4d6d2a4b2c"
server_key_version: "encryptionkey@0"
I20260927 13:54:53.106261 9953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:53.119313 9953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:53.124610 9953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:53.124895 9953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:53.144624 9953 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41377
I20260927 13:54:53.144704 10019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41377 every 8 connection(s)
I20260927 13:54:53.146083 9953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:53.150957 10020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:53.152987 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 9953
I20260927 13:54:53.153200 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:53.153801 21013 external_mini_cluster.cc:1428] Setting key 61e767f3fb77c30efd7cfc6562d046a2
I20260927 13:54:53.167382 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282: Bootstrap starting.
Sep 27 13:54:53 dist-test-slave-q25r krb5kdc[9940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 13:54:53.171725 10020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:53.173267 10020 log.cc:824] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:53.177286 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282: No bootstrap required, opened a new log
I20260927 13:54:53.180264 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:53.156230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:49835 (local address 127.0.0.1:41377)
0927 13:54:53.157396 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:53.157411 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:53.157464 (+ 53us) server_negotiation.cc:407] Connection header received
0927 13:54:53.159473 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:53.159503 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:53.160092 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:53.160578 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:53.162170 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:53.163594 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:53.165006 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:53.165751 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:53.165933 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:53.170903 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:53.170914 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:53.170935 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:53.170989 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:53.175065 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:53.176076 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:53.176085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:53.176093 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:53.176286 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:53.176805 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:53.176809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:53.176811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:53.177606 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:53.178026 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:53.178679 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 13:54:53.179043 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":474,"threads_started":1}
I20260927 13:54:53.184770 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:53.186820 10020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d47d07d7bc4f4991afc46e469ef282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } }
I20260927 13:54:53.187345 10020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:53.187448 10020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d47d07d7bc4f4991afc46e469ef282, State: Initialized, Role: FOLLOWER
I20260927 13:54:53.188201 10020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d47d07d7bc4f4991afc46e469ef282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } }
I20260927 13:54:53.188511 10020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:53.188666 10020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:53.188845 10020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:53.192534 10020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d47d07d7bc4f4991afc46e469ef282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } }
I20260927 13:54:53.193360 10020 leader_election.cc:302] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83d47d07d7bc4f4991afc46e469ef282; no voters:
I20260927 13:54:53.196271 10020 leader_election.cc:288] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:53.196507 10030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:53.197938 10030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [term 1 LEADER]: Becoming Leader. State: Replica: 83d47d07d7bc4f4991afc46e469ef282, State: Running, Role: LEADER
I20260927 13:54:53.198678 10030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d47d07d7bc4f4991afc46e469ef282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } }
I20260927 13:54:53.199679 10020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:53.204205 10031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83d47d07d7bc4f4991afc46e469ef282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d47d07d7bc4f4991afc46e469ef282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } } }
I20260927 13:54:53.204896 10031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:53.205582 10032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d47d07d7bc4f4991afc46e469ef282. Latest consensus state: current_term: 1 leader_uuid: "83d47d07d7bc4f4991afc46e469ef282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d47d07d7bc4f4991afc46e469ef282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } } }
I20260927 13:54:53.205940 10032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282 [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:53.210431 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:53.189638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:37167 (local address 127.0.0.1:41377)
0927 13:54:53.189884 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:53.189890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:53.190320 (+ 430us) server_negotiation.cc:407] Connection header received
0927 13:54:53.190600 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:53.190610 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:53.190695 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:53.190818 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:53.192805 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:53.193721 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:53.195915 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:53.196615 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:53.196797 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:53.199781 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:53.199788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:53.199792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:53.199841 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:53.204611 (+ 4770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:53.206110 (+ 1499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:53.206116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:53.206120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:53.206392 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:53.207210 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:53.207217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:53.207222 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:53.207783 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:53.208085 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:53.209713 (+ 1628us) server_negotiation.cc:299] Negotiation successful
0927 13:54:53.209995 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 13:54:53.217119 10036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:53.224148 10036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:53.236083 10036 catalog_manager.cc:1347] Generated new cluster ID: a28a3b12f81e4266a13ec19ceb2c51d5
I20260927 13:54:53.236246 10036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:53.261073 10036 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:53.263515 10036 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:53.279073 10036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83d47d07d7bc4f4991afc46e469ef282: Generated new TSK 0
I20260927 13:54:53.280743 10036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:53.292815 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 9953
I20260927 13:54:53.294694 10015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:53.390385 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 9953
W20260927 13:54:53.408936 9938 connection.cc:535] client connection to 127.0.0.1:41377 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41377): Connection reset by peer (error 104)
2026-09-27T13:54:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:53 dist-test-slave-q25r krb5kdc[10052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:53 dist-test-slave-q25r krb5kdc[10052](info): set up 2 sockets
Sep 27 13:54:53 dist-test-slave-q25r krb5kdc[10052](info): commencing operation
krb5kdc: starting...
W20260927 13:54:54.440608 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:54 dist-test-slave-q25r krb5kdc[10052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:54.701675 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:55.035602 10065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:55.035921 10065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:55.036047 10065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:55.043514 10065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:55.043658 10065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:55.043741 10065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:55.043823 10065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:55.043883 10065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:55.054265 10065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:55.055634 10065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:55.057060 10065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:55.069751 10073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:55.069763 10071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:55.070122 10070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:55.071092 10065 server_base.cc:1034] running on GCE node
I20260927 13:54:55.072149 10065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:55.074090 10065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:55.075345 10065 hybrid_clock.cc:648] HybridClock initialized: now 1790517295075261 us; error 66 us; skew 500 ppm
I20260927 13:54:55.075757 10065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:55 dist-test-slave-q25r krb5kdc[10052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:55.079584 10065 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:55.081799 10065 webserver.cc:460] Webserver started at http://127.0.0.1:46769/ using document root <none> and password file <none>
I20260927 13:54:55.082756 10065 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:55.082891 10065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:55.083243 10065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:55.087152 10065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "bbcf3353d6194c65852a50baf98f588f"
format_stamp: "Formatted at 2026-09-27 13:54:55 on dist-test-slave-q25r"
server_key: "bb1e0aa6bf964c4abc4861be98d34e04"
server_key_iv: "5313c87a91a3014c73cacbd393d8ab14"
server_key_version: "encryptionkey@0"
I20260927 13:54:55.088081 10065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "bbcf3353d6194c65852a50baf98f588f"
format_stamp: "Formatted at 2026-09-27 13:54:55 on dist-test-slave-q25r"
server_key: "bb1e0aa6bf964c4abc4861be98d34e04"
server_key_iv: "5313c87a91a3014c73cacbd393d8ab14"
server_key_version: "encryptionkey@0"
I20260927 13:54:55.094624 10065 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 13:54:55.099086 10080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:55.100973 10065 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 13:54:55.101186 10065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "bbcf3353d6194c65852a50baf98f588f"
format_stamp: "Formatted at 2026-09-27 13:54:55 on dist-test-slave-q25r"
server_key: "bb1e0aa6bf964c4abc4861be98d34e04"
server_key_iv: "5313c87a91a3014c73cacbd393d8ab14"
server_key_version: "encryptionkey@0"
I20260927 13:54:55.101419 10065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:55.117340 10065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:55.122673 10065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:55.123018 10065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:55.144838 10065 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45205
I20260927 13:54:55.144869 10131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45205 every 8 connection(s)
I20260927 13:54:55.146553 10065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:55.150810 10132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:55.152940 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 10065
I20260927 13:54:55.153126 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:55.153592 21013 external_mini_cluster.cc:1428] Setting key 9134208c95bc666096624b94b2f9642e
I20260927 13:54:55.164933 10132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f: Bootstrap starting.
I20260927 13:54:55.168594 10132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:55.169826 10132 log.cc:824] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:55.170935 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:55.155978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:60615 (local address 127.0.0.1:45205)
0927 13:54:55.156999 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:55.157019 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:55.157082 (+ 63us) server_negotiation.cc:407] Connection header received
0927 13:54:55.159443 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:55.159475 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:55.160059 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:55.160587 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:55.161319 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:55.163422 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:55.165166 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:55.166101 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:55.166336 (+ 235us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:55.167916 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:55.167925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:55.167957 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:55.168735 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:55.168872 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:55.168909 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:55.169086 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:55.169375 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 13:54:55.169592 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":393,"threads_started":1}
I20260927 13:54:55.173305 10132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f: No bootstrap required, opened a new log
I20260927 13:54:55.175824 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:55.181432 10132 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcf3353d6194c65852a50baf98f588f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } }
I20260927 13:54:55.181967 10132 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:55.182088 10132 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbcf3353d6194c65852a50baf98f588f, State: Initialized, Role: FOLLOWER
I20260927 13:54:55.182839 10132 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcf3353d6194c65852a50baf98f588f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } }
I20260927 13:54:55.183092 10132 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:55.183251 10132 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:55.183418 10132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:55.186466 10132 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcf3353d6194c65852a50baf98f588f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } }
I20260927 13:54:55.187075 10132 leader_election.cc:302] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbcf3353d6194c65852a50baf98f588f; no voters:
I20260927 13:54:55.187819 10132 leader_election.cc:288] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:55.188071 10142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:55.188963 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:55.180831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:41041 (local address 127.0.0.1:45205)
0927 13:54:55.181065 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:55.181072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:55.181470 (+ 398us) server_negotiation.cc:407] Connection header received
0927 13:54:55.181708 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:55.181712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:55.181819 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:55.181924 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:55.182643 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:55.183700 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:55.185650 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:55.186165 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:55.186265 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:55.187653 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:55.187660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:55.187665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:55.187695 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:55.187782 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:55.187818 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:55.187948 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:55.188465 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 13:54:55.188547 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 13:54:55.189563 10142 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [term 1 LEADER]: Becoming Leader. State: Replica: bbcf3353d6194c65852a50baf98f588f, State: Running, Role: LEADER
I20260927 13:54:55.190177 10132 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:55.190291 10142 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcf3353d6194c65852a50baf98f588f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } }
I20260927 13:54:55.194397 10143 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbcf3353d6194c65852a50baf98f588f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcf3353d6194c65852a50baf98f588f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } } }
I20260927 13:54:55.194911 10143 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:55.194341 10144 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbcf3353d6194c65852a50baf98f588f. Latest consensus state: current_term: 1 leader_uuid: "bbcf3353d6194c65852a50baf98f588f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcf3353d6194c65852a50baf98f588f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } } }
I20260927 13:54:55.196517 10144 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:55.198999 10152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:55.204785 10152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:55.216560 10152 catalog_manager.cc:1347] Generated new cluster ID: a11a629abc9e4bbf88c60e4f5529de42
I20260927 13:54:55.216976 10152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:55.261777 10152 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:55.263952 10152 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:55.280135 10152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbcf3353d6194c65852a50baf98f588f: Generated new TSK 0
I20260927 13:54:55.281392 10152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:55.295447 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10065
W20260927 13:54:55.299388 10127 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.34:60615
W20260927 13:54:55.300781 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10065: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:54:55.301047 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 10065
W20260927 13:54:55.320698 10050 connection.cc:535] client connection to 127.0.0.1:45205 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45205): Connection reset by peer (error 104)
2026-09-27T13:54:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:55 dist-test-slave-q25r krb5kdc[10163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:55 dist-test-slave-q25r krb5kdc[10163](info): set up 2 sockets
Sep 27 13:54:55 dist-test-slave-q25r krb5kdc[10163](info): commencing operation
krb5kdc: starting...
W20260927 13:54:56.364873 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.007s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:56 dist-test-slave-q25r krb5kdc[10163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:54:56.667479 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:57.002463 10176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:57.002789 10176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:57.002897 10176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:57.010576 10176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:57.010718 10176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:57.010767 10176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:57.010845 10176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:57.010919 10176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:57.021379 10176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35607
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:57.022742 10176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:57.024097 10176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:57.036792 10182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:57.036880 10181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:57.037395 10184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:57.042551 10176 server_base.cc:1034] running on GCE node
I20260927 13:54:57.043540 10176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:57.045248 10176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:57.046497 10176 hybrid_clock.cc:648] HybridClock initialized: now 1790517297046450 us; error 49 us; skew 500 ppm
I20260927 13:54:57.046888 10176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:57 dist-test-slave-q25r krb5kdc[10163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:57.057149 10176 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:54:57.059015 10176 webserver.cc:460] Webserver started at http://127.0.0.1:37629/ using document root <none> and password file <none>
I20260927 13:54:57.059792 10176 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:57.059918 10176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:57.060238 10176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:57.063441 10176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "6681cc523981473c9629e07e91cc4b3e"
format_stamp: "Formatted at 2026-09-27 13:54:57 on dist-test-slave-q25r"
server_key: "d4ccc76ff0ff866b872628f81d6ccbd4"
server_key_iv: "e39f53443d052d6e2dd8337e1cc91db8"
server_key_version: "encryptionkey@0"
I20260927 13:54:57.064229 10176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "6681cc523981473c9629e07e91cc4b3e"
format_stamp: "Formatted at 2026-09-27 13:54:57 on dist-test-slave-q25r"
server_key: "d4ccc76ff0ff866b872628f81d6ccbd4"
server_key_iv: "e39f53443d052d6e2dd8337e1cc91db8"
server_key_version: "encryptionkey@0"
I20260927 13:54:57.069980 10176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 13:54:57.073897 10191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:57.075532 10176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:54:57.075743 10176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "6681cc523981473c9629e07e91cc4b3e"
format_stamp: "Formatted at 2026-09-27 13:54:57 on dist-test-slave-q25r"
server_key: "d4ccc76ff0ff866b872628f81d6ccbd4"
server_key_iv: "e39f53443d052d6e2dd8337e1cc91db8"
server_key_version: "encryptionkey@0"
I20260927 13:54:57.075946 10176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:57.082464 10176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:57.087085 10176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:57.087410 10176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:57.107014 10176 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35607
I20260927 13:54:57.107101 10242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35607 every 8 connection(s)
I20260927 13:54:57.108587 10176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:57.113404 10243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:54:57.117784 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 10176
I20260927 13:54:57.118005 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:57.118634 21013 external_mini_cluster.cc:1428] Setting key fee6ed45dad5ac41ad0c02d23746e1fe
I20260927 13:54:57.127971 10246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:57.121070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:45531 (local address 127.0.0.1:35607)
0927 13:54:57.122115 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:57.122133 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:57.122183 (+ 50us) server_negotiation.cc:407] Connection header received
0927 13:54:57.124175 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:57.124202 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:57.124701 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:57.125099 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:57.126258 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:57.126264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:57.126276 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:57.126292 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:57.126356 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:57.126381 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:57.126610 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:57.126752 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 13:54:57.126940 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":451,"threads_started":1}
I20260927 13:54:57.128937 10243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e: Bootstrap starting.
I20260927 13:54:57.132756 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:57.133298 10243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:57.134521 10243 log.cc:824] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:57.138800 10243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e: No bootstrap required, opened a new log
I20260927 13:54:57.140269 10246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:57.137778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.34:47125 (local address 127.0.0.1:35607)
0927 13:54:57.137963 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:57.137970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:57.138406 (+ 436us) server_negotiation.cc:407] Connection header received
0927 13:54:57.138583 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:57.138587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:57.138639 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:57.138725 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:57.139634 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:57.139640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:57.139645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:54:57.139663 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:57.139705 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:54:57.139720 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:57.139820 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:57.139927 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 13:54:57.139989 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:54:57.147693 10243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681cc523981473c9629e07e91cc4b3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35607 } }
I20260927 13:54:57.148196 10243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:57.148313 10243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6681cc523981473c9629e07e91cc4b3e, State: Initialized, Role: FOLLOWER
I20260927 13:54:57.149044 10243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681cc523981473c9629e07e91cc4b3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35607 } }
I20260927 13:54:57.149308 10243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:57.149461 10243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:57.149631 10243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:57.152233 10243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681cc523981473c9629e07e91cc4b3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35607 } }
I20260927 13:54:57.152942 10243 leader_election.cc:302] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6681cc523981473c9629e07e91cc4b3e; no voters:
I20260927 13:54:57.153652 10243 leader_election.cc:288] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:57.153828 10253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:57.154860 10253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [term 1 LEADER]: Becoming Leader. State: Replica: 6681cc523981473c9629e07e91cc4b3e, State: Running, Role: LEADER
I20260927 13:54:57.155495 10253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681cc523981473c9629e07e91cc4b3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35607 } }
I20260927 13:54:57.156422 10243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:57.158246 10255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6681cc523981473c9629e07e91cc4b3e. Latest consensus state: current_term: 1 leader_uuid: "6681cc523981473c9629e07e91cc4b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681cc523981473c9629e07e91cc4b3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35607 } } }
I20260927 13:54:57.158985 10255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:57.158607 10254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6681cc523981473c9629e07e91cc4b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681cc523981473c9629e07e91cc4b3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35607 } } }
I20260927 13:54:57.159818 10254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:57.161130 10258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:57.167101 10258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:57.177851 10258 catalog_manager.cc:1347] Generated new cluster ID: 27434a1a5de44d5eb6592af53a0fc4e0
I20260927 13:54:57.178025 10258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:57.235282 10258 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:57.235903 10258 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:57.246322 10258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6681cc523981473c9629e07e91cc4b3e: Generated new TSK 0
I20260927 13:54:57.247606 10258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:57.253444 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10176
W20260927 13:54:57.254760 10238 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.34:45531
W20260927 13:54:57.255713 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10176: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:54:57.255945 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 10176
2026-09-27T13:54:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:57 dist-test-slave-q25r krb5kdc[10275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:57 dist-test-slave-q25r krb5kdc[10275](info): set up 2 sockets
Sep 27 13:54:57 dist-test-slave-q25r krb5kdc[10275](info): commencing operation
krb5kdc: starting...
W20260927 13:54:58.300477 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:54:58 dist-test-slave-q25r krb5kdc[10275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:54:58.556011 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41439
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:54:58.881877 10288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:54:58.882179 10288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:54:58.882314 10288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:54:58.889683 10288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:54:58.889811 10288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:54:58.889889 10288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:54:58.889971 10288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:54:58.890029 10288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:54:58.900434 10288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:41439
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:54:58.901799 10288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:54:58.903103 10288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:54:58.914614 10294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:58.914606 10293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:54:58.914758 10296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:54:58.915485 10288 server_base.cc:1034] running on GCE node
I20260927 13:54:58.916231 10288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:54:58.917701 10288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:54:58.918905 10288 hybrid_clock.cc:648] HybridClock initialized: now 1790517298918861 us; error 37 us; skew 500 ppm
I20260927 13:54:58.919269 10288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:54:58 dist-test-slave-q25r krb5kdc[10275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517298, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:54:58.922197 10288 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:54:58.923718 10288 webserver.cc:460] Webserver started at http://127.20.133.126:44481/ using document root <none> and password file <none>
I20260927 13:54:58.924431 10288 fs_manager.cc:359] Metadata directory not provided
I20260927 13:54:58.924566 10288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:54:58.924851 10288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:54:58.927902 10288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "6d9c90fe1c134fea8c9860491beff47c"
format_stamp: "Formatted at 2026-09-27 13:54:58 on dist-test-slave-q25r"
server_key: "de4ecf7a77f55cc6f37d40311ace975f"
server_key_iv: "1e9e592e4f1e4e2815d10dd65528cf15"
server_key_version: "encryptionkey@0"
I20260927 13:54:58.928735 10288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "6d9c90fe1c134fea8c9860491beff47c"
format_stamp: "Formatted at 2026-09-27 13:54:58 on dist-test-slave-q25r"
server_key: "de4ecf7a77f55cc6f37d40311ace975f"
server_key_iv: "1e9e592e4f1e4e2815d10dd65528cf15"
server_key_version: "encryptionkey@0"
I20260927 13:54:58.934844 10288 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 13:54:58.938472 10303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:54:58.939877 10288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:54:58.940042 10288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "6d9c90fe1c134fea8c9860491beff47c"
format_stamp: "Formatted at 2026-09-27 13:54:58 on dist-test-slave-q25r"
server_key: "de4ecf7a77f55cc6f37d40311ace975f"
server_key_iv: "1e9e592e4f1e4e2815d10dd65528cf15"
server_key_version: "encryptionkey@0"
I20260927 13:54:58.940232 10288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:54:58.954493 10288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:54:58.959239 10288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:54:58.959545 10288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:54:58.977250 10288 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:41439
I20260927 13:54:58.977283 10354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:41439 every 8 connection(s)
I20260927 13:54:58.978755 10288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:54:58.980544 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 10288
I20260927 13:54:58.980703 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:54:58.981158 21013 external_mini_cluster.cc:1428] Setting key f464e5505ddf76ecd9576a1b30e4bd75
I20260927 13:54:58.982869 10355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:54:58 dist-test-slave-q25r krb5kdc[10275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:54:58.996460 10355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c: Bootstrap starting.
I20260927 13:54:59.000321 10355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c: Neither blocks nor log segments found. Creating new log.
I20260927 13:54:59.001689 10355 log.cc:824] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c: Log is configured to *not* fsync() on all Append() calls
I20260927 13:54:59.005095 10355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c: No bootstrap required, opened a new log
I20260927 13:54:59.005704 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:58.982768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47711 (local address 127.20.133.126:41439)
0927 13:54:58.983578 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:58.983594 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:58.983643 (+ 49us) server_negotiation.cc:407] Connection header received
0927 13:54:58.985821 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:58.985849 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:58.986394 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:58.986872 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:58.988280 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:58.990321 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:58.991631 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:58.992679 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:58.992867 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:58.997299 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:58.997309 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:58.997326 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:58.997369 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:59.000781 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:59.001985 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:59.001992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:59.002000 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:59.002194 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:59.002738 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:59.002742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:59.002744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:59.003332 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:59.003678 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:59.004213 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 13:54:59.004590 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":343,"threads_started":1}
I20260927 13:54:59.010303 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:54:59.014493 10355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9c90fe1c134fea8c9860491beff47c" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41439 } }
I20260927 13:54:59.015092 10355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:54:59.015296 10355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d9c90fe1c134fea8c9860491beff47c, State: Initialized, Role: FOLLOWER
I20260927 13:54:59.015986 10355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9c90fe1c134fea8c9860491beff47c" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41439 } }
I20260927 13:54:59.016227 10355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:54:59.016400 10355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:54:59.016567 10355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:54:59.018978 10355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9c90fe1c134fea8c9860491beff47c" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41439 } }
I20260927 13:54:59.019556 10355 leader_election.cc:302] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d9c90fe1c134fea8c9860491beff47c; no voters:
I20260927 13:54:59.020265 10355 leader_election.cc:288] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:54:59.020540 10365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:54:59.022284 10365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [term 1 LEADER]: Becoming Leader. State: Replica: 6d9c90fe1c134fea8c9860491beff47c, State: Running, Role: LEADER
I20260927 13:54:59.022966 10365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9c90fe1c134fea8c9860491beff47c" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41439 } }
I20260927 13:54:59.023159 10355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:54:59.027598 10366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d9c90fe1c134fea8c9860491beff47c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9c90fe1c134fea8c9860491beff47c" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41439 } } }
I20260927 13:54:59.028047 10366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:59.030552 10367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d9c90fe1c134fea8c9860491beff47c. Latest consensus state: current_term: 1 leader_uuid: "6d9c90fe1c134fea8c9860491beff47c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9c90fe1c134fea8c9860491beff47c" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41439 } } }
I20260927 13:54:59.030942 10367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c [sys.catalog]: This master's current role is: LEADER
I20260927 13:54:59.034978 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:54:59.014983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37029 (local address 127.20.133.126:41439)
0927 13:54:59.015220 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 13:54:59.015227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:54:59.015625 (+ 398us) server_negotiation.cc:407] Connection header received
0927 13:54:59.015807 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:54:59.015812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:54:59.015878 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:54:59.015960 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:54:59.017671 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:59.018470 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:59.019822 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:54:59.020646 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:54:59.021511 (+ 865us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:54:59.024066 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:54:59.024073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:54:59.024079 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:54:59.024126 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:54:59.029215 (+ 5089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:59.031302 (+ 2087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:59.031308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:59.031313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:59.031524 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:54:59.032191 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:54:59.032197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:54:59.032201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:54:59.032690 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:54:59.033031 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 13:54:59.034329 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0927 13:54:59.034573 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 13:54:59.037918 10372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:54:59.043308 10372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:54:59.054154 10372 catalog_manager.cc:1347] Generated new cluster ID: 064d2a413a0b4d71b1737ef9960fb531
I20260927 13:54:59.054291 10372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:54:59.082823 10372 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:54:59.084262 10372 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:54:59.097287 10372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d9c90fe1c134fea8c9860491beff47c: Generated new TSK 0
I20260927 13:54:59.098256 10372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:54:59.112569 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10288
I20260927 13:54:59.113885 10350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:54:59.206810 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 10288
W20260927 13:54:59.220901 10273 connection.cc:535] client connection to 127.20.133.126:41439 recv error: Network error: failed to read from TLS socket (remote: 127.20.133.126:41439): Connection reset by peer (error 104)
2026-09-27T13:54:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1946 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:54:59 dist-test-slave-q25r krb5kdc[10387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:54:59 dist-test-slave-q25r krb5kdc[10387](info): set up 2 sockets
Sep 27 13:54:59 dist-test-slave-q25r krb5kdc[10387](info): commencing operation
krb5kdc: starting...
W20260927 13:55:00.225005 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:55:00 dist-test-slave-q25r krb5kdc[10387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:55:00.523602 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41717
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:41717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:00.917801 10400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:00.918170 10400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:00.918280 10400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:00.926762 10400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:00.926946 10400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:00.926998 10400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:00.927069 10400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:00.927119 10400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:00.938967 10400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:41717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:00.940764 10400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:00.942345 10400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:00.958421 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:00.958504 10408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:00.958467 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:00.959468 10400 server_base.cc:1034] running on GCE node
I20260927 13:55:00.960443 10400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:00.962057 10400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:00.963304 10400 hybrid_clock.cc:648] HybridClock initialized: now 1790517300963259 us; error 46 us; skew 500 ppm
I20260927 13:55:00.963761 10400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:00 dist-test-slave-q25r krb5kdc[10387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517300, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:00.967115 10400 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:55:00.968925 10400 webserver.cc:460] Webserver started at http://127.20.133.126:45261/ using document root <none> and password file <none>
I20260927 13:55:00.969674 10400 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:00.969784 10400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:00.970043 10400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:00.973284 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "81abba6ac9b64b79853c64fc3a323c8a"
format_stamp: "Formatted at 2026-09-27 13:55:00 on dist-test-slave-q25r"
server_key: "7d092093aec73f08b726b51066da6312"
server_key_iv: "a7be73593406ef158b2af6b2d5f9e648"
server_key_version: "encryptionkey@0"
I20260927 13:55:00.974069 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "81abba6ac9b64b79853c64fc3a323c8a"
format_stamp: "Formatted at 2026-09-27 13:55:00 on dist-test-slave-q25r"
server_key: "7d092093aec73f08b726b51066da6312"
server_key_iv: "a7be73593406ef158b2af6b2d5f9e648"
server_key_version: "encryptionkey@0"
I20260927 13:55:00.980047 10400 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 13:55:00.984368 10415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:00.986375 10400 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:55:00.986613 10400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "81abba6ac9b64b79853c64fc3a323c8a"
format_stamp: "Formatted at 2026-09-27 13:55:00 on dist-test-slave-q25r"
server_key: "7d092093aec73f08b726b51066da6312"
server_key_iv: "a7be73593406ef158b2af6b2d5f9e648"
server_key_version: "encryptionkey@0"
I20260927 13:55:00.986846 10400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:01.013043 10400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:01.017706 10400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:01.018010 10400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:01.038092 10400 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:41717
I20260927 13:55:01.038112 10466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:41717 every 8 connection(s)
I20260927 13:55:01.039536 10400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:01.043468 10467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:01.045150 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 10400
I20260927 13:55:01.045307 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:55:01.045773 21013 external_mini_cluster.cc:1428] Setting key 57230ab984ed15229d0c9f3a4cf04938
I20260927 13:55:01.055850 10467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a: Bootstrap starting.
I20260927 13:55:01.059751 10467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:01.060643 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:01.047821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60957 (local address 127.20.133.126:41717)
0927 13:55:01.048750 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:01.048762 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:01.048812 (+ 50us) server_negotiation.cc:407] Connection header received
0927 13:55:01.050630 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:01.050654 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:01.051080 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:01.051440 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:01.052129 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:01.053633 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:01.055466 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:01.056428 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:01.056594 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:01.057982 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:01.057991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:01.058007 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:55:01.058597 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:01.058689 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:55:01.058718 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:01.058852 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:01.059172 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 13:55:01.059369 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":431,"threads_started":1}
I20260927 13:55:01.061028 10467 log.cc:824] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:01.064324 10467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a: No bootstrap required, opened a new log
I20260927 13:55:01.065047 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:55:01.072631 10467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81abba6ac9b64b79853c64fc3a323c8a" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41717 } }
I20260927 13:55:01.073093 10467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:01.073189 10467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81abba6ac9b64b79853c64fc3a323c8a, State: Initialized, Role: FOLLOWER
I20260927 13:55:01.073849 10467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81abba6ac9b64b79853c64fc3a323c8a" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41717 } }
I20260927 13:55:01.074112 10467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:55:01.074270 10467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:55:01.074432 10467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:01.076097 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:01.069780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51973 (local address 127.20.133.126:41717)
0927 13:55:01.069970 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:01.069976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:01.070350 (+ 374us) server_negotiation.cc:407] Connection header received
0927 13:55:01.070545 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:01.070550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:01.070624 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:01.070725 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:01.071249 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:01.072012 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:01.073464 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:01.073941 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:01.074077 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:01.075304 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:01.075309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:01.075312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:55:01.075335 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:01.075407 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:55:01.075425 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:01.075537 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:01.075754 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 13:55:01.075816 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:55:01.077101 10467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81abba6ac9b64b79853c64fc3a323c8a" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41717 } }
I20260927 13:55:01.077692 10467 leader_election.cc:302] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81abba6ac9b64b79853c64fc3a323c8a; no voters:
I20260927 13:55:01.078289 10467 leader_election.cc:288] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:55:01.078423 10477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:01.079119 10477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [term 1 LEADER]: Becoming Leader. State: Replica: 81abba6ac9b64b79853c64fc3a323c8a, State: Running, Role: LEADER
I20260927 13:55:01.079689 10477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81abba6ac9b64b79853c64fc3a323c8a" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41717 } }
I20260927 13:55:01.080394 10467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:01.083258 10479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81abba6ac9b64b79853c64fc3a323c8a. Latest consensus state: current_term: 1 leader_uuid: "81abba6ac9b64b79853c64fc3a323c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81abba6ac9b64b79853c64fc3a323c8a" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41717 } } }
I20260927 13:55:01.083781 10479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:01.084728 10484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:01.084539 10478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81abba6ac9b64b79853c64fc3a323c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81abba6ac9b64b79853c64fc3a323c8a" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41717 } } }
I20260927 13:55:01.084921 10478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:01.092350 10484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:55:01.103679 10484 catalog_manager.cc:1347] Generated new cluster ID: 89d2f385d17b47949b0cdceb10de086f
I20260927 13:55:01.103813 10484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:01.124408 10484 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:01.126441 10484 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:01.137019 10484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81abba6ac9b64b79853c64fc3a323c8a: Generated new TSK 0
I20260927 13:55:01.137905 10484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:55:01.144105 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10400
W20260927 13:55:01.145884 10462 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60957
W20260927 13:55:01.146886 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10400: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:55:01.147029 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 10400
W20260927 13:55:01.161613 10385 connection.cc:535] client connection to 127.20.133.126:41717 recv error: Network error: failed to read from TLS socket (remote: 127.20.133.126:41717): Connection reset by peer (error 104)
2026-09-27T13:55:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:01 dist-test-slave-q25r krb5kdc[10498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:01 dist-test-slave-q25r krb5kdc[10498](info): set up 2 sockets
Sep 27 13:55:01 dist-test-slave-q25r krb5kdc[10498](info): commencing operation
krb5kdc: starting...
W20260927 13:55:02.184818 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:55:02 dist-test-slave-q25r krb5kdc[10498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:55:02.487722 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:34847
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:34847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:02.848600 10511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:02.848913 10511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:02.849041 10511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:02.856681 10511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:02.856823 10511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:02.856881 10511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:02.857071 10511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:02.857192 10511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:02.867770 10511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:34847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:02.869203 10511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:02.870524 10511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:02.883420 10519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:02.883498 10516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:02.883677 10517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:02.884842 10511 server_base.cc:1034] running on GCE node
I20260927 13:55:02.885696 10511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:02.887324 10511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:02.888610 10511 hybrid_clock.cc:648] HybridClock initialized: now 1790517302888522 us; error 88 us; skew 500 ppm
I20260927 13:55:02.889010 10511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:02 dist-test-slave-q25r krb5kdc[10498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517302, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:02.896086 10511 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:55:02.897971 10511 webserver.cc:460] Webserver started at http://127.20.133.126:46821/ using document root <none> and password file <none>
I20260927 13:55:02.898720 10511 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:02.898849 10511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:02.899159 10511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:02.902295 10511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "08c333047b784ef088b175fe3c2b91cf"
format_stamp: "Formatted at 2026-09-27 13:55:02 on dist-test-slave-q25r"
server_key: "7f1306e054c13c733d9be51686431f66"
server_key_iv: "faaa102c0ccff38cbcc410315c6f805c"
server_key_version: "encryptionkey@0"
I20260927 13:55:02.903081 10511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "08c333047b784ef088b175fe3c2b91cf"
format_stamp: "Formatted at 2026-09-27 13:55:02 on dist-test-slave-q25r"
server_key: "7f1306e054c13c733d9be51686431f66"
server_key_iv: "faaa102c0ccff38cbcc410315c6f805c"
server_key_version: "encryptionkey@0"
I20260927 13:55:02.909117 10511 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 13:55:02.913233 10526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:02.914855 10511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 13:55:02.915050 10511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "08c333047b784ef088b175fe3c2b91cf"
format_stamp: "Formatted at 2026-09-27 13:55:02 on dist-test-slave-q25r"
server_key: "7f1306e054c13c733d9be51686431f66"
server_key_iv: "faaa102c0ccff38cbcc410315c6f805c"
server_key_version: "encryptionkey@0"
I20260927 13:55:02.915251 10511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:02.921300 10511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:02.925815 10511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:02.926112 10511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:02.945348 10511 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:34847
I20260927 13:55:02.945377 10577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:34847 every 8 connection(s)
I20260927 13:55:02.946768 10511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:02.949463 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 10511
I20260927 13:55:02.949697 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:55:02.950307 21013 external_mini_cluster.cc:1428] Setting key 55392cca7eeb165917b1cf3cac69354c
I20260927 13:55:02.952154 10578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:02.959937 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:02.952763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42017 (local address 127.20.133.126:34847)
0927 13:55:02.953857 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:02.953873 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:02.953929 (+ 56us) server_negotiation.cc:407] Connection header received
0927 13:55:02.955732 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:02.955764 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:02.956236 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:02.956757 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:02.958043 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:02.958049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:02.958069 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:55:02.958099 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:02.958193 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:55:02.958218 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:02.958492 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:02.958661 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 13:55:02.958873 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":455,"threads_started":1}
I20260927 13:55:02.964461 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:55:02.968170 10578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf: Bootstrap starting.
I20260927 13:55:02.972240 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:02.969335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53069 (local address 127.20.133.126:34847)
0927 13:55:02.969510 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:02.969516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:02.970090 (+ 574us) server_negotiation.cc:407] Connection header received
0927 13:55:02.970281 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:02.970285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:02.970348 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:02.970447 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:02.971586 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:02.971591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:02.971594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:55:02.971612 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:02.971657 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:55:02.971674 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:02.971788 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:02.971896 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 13:55:02.971962 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:55:02.972972 10578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:02.974344 10578 log.cc:824] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:02.977792 10578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf: No bootstrap required, opened a new log
I20260927 13:55:02.986907 10578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c333047b784ef088b175fe3c2b91cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34847 } }
I20260927 13:55:02.987392 10578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:02.987500 10578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c333047b784ef088b175fe3c2b91cf, State: Initialized, Role: FOLLOWER
I20260927 13:55:02.988180 10578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c333047b784ef088b175fe3c2b91cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34847 } }
I20260927 13:55:02.988453 10578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:55:02.988607 10578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:55:02.988786 10578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:02.991223 10578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c333047b784ef088b175fe3c2b91cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34847 } }
I20260927 13:55:02.991863 10578 leader_election.cc:302] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08c333047b784ef088b175fe3c2b91cf; no voters:
I20260927 13:55:02.992674 10578 leader_election.cc:288] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:55:02.992900 10588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:02.993769 10588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [term 1 LEADER]: Becoming Leader. State: Replica: 08c333047b784ef088b175fe3c2b91cf, State: Running, Role: LEADER
I20260927 13:55:02.994501 10588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c333047b784ef088b175fe3c2b91cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34847 } }
I20260927 13:55:02.995102 10578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:02.997896 10589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08c333047b784ef088b175fe3c2b91cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c333047b784ef088b175fe3c2b91cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34847 } } }
I20260927 13:55:02.998126 10590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08c333047b784ef088b175fe3c2b91cf. Latest consensus state: current_term: 1 leader_uuid: "08c333047b784ef088b175fe3c2b91cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c333047b784ef088b175fe3c2b91cf" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 34847 } } }
I20260927 13:55:02.998606 10589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:02.998646 10590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:03.001171 10596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:03.006991 10596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:55:03.018342 10596 catalog_manager.cc:1347] Generated new cluster ID: cd36050a43874cd1ba7a78295a68dbb1
I20260927 13:55:03.018543 10596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:03.041201 10596 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:03.041731 10596 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:03.063057 10596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08c333047b784ef088b175fe3c2b91cf: Generated new TSK 0
I20260927 13:55:03.063974 10596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:55:03.071034 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10511
W20260927 13:55:03.072391 10573 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42017
W20260927 13:55:03.073290 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10511: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:55:03.073453 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 10511
2026-09-27T13:55:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:03 dist-test-slave-q25r krb5kdc[10609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:03 dist-test-slave-q25r krb5kdc[10609](info): set up 2 sockets
Sep 27 13:55:03 dist-test-slave-q25r krb5kdc[10609](info): commencing operation
krb5kdc: starting...
W20260927 13:55:04.100906 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:55:04 dist-test-slave-q25r krb5kdc[10609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:55:04.365664 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:33785
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:33785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:04.708132 10622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:04.708523 10622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:04.708666 10622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:04.716604 10622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:04.716746 10622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:04.716802 10622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:04.716883 10622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:04.716943 10622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:04.727355 10622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:33785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:33785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:04.728813 10622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:04.730113 10622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:04.741427 10628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:04.741586 10627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:04.741559 10630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:04.742467 10622 server_base.cc:1034] running on GCE node
I20260927 13:55:04.743274 10622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:04.744719 10622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:04.745954 10622 hybrid_clock.cc:648] HybridClock initialized: now 1790517304745897 us; error 46 us; skew 500 ppm
I20260927 13:55:04.746296 10622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:04 dist-test-slave-q25r krb5kdc[10609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517304, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:04.749374 10622 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:55:04.750857 10622 webserver.cc:460] Webserver started at http://127.20.133.126:44783/ using document root <none> and password file <none>
I20260927 13:55:04.751578 10622 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:04.751688 10622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:04.751981 10622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:04.755120 10622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "2f182d9fafb6407c9bda807e3b629a3b"
format_stamp: "Formatted at 2026-09-27 13:55:04 on dist-test-slave-q25r"
server_key: "deb4669411c0fe002e6666bafd1ae02d"
server_key_iv: "72e3efee03005014f424cff6e2565ea5"
server_key_version: "encryptionkey@0"
I20260927 13:55:04.755889 10622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "2f182d9fafb6407c9bda807e3b629a3b"
format_stamp: "Formatted at 2026-09-27 13:55:04 on dist-test-slave-q25r"
server_key: "deb4669411c0fe002e6666bafd1ae02d"
server_key_iv: "72e3efee03005014f424cff6e2565ea5"
server_key_version: "encryptionkey@0"
I20260927 13:55:04.761372 10622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 13:55:04.765177 10637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:04.766520 10622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:55:04.766709 10622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "2f182d9fafb6407c9bda807e3b629a3b"
format_stamp: "Formatted at 2026-09-27 13:55:04 on dist-test-slave-q25r"
server_key: "deb4669411c0fe002e6666bafd1ae02d"
server_key_iv: "72e3efee03005014f424cff6e2565ea5"
server_key_version: "encryptionkey@0"
I20260927 13:55:04.766906 10622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:04.798076 10622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:04.803511 10622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:04.803826 10622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:04.821444 10622 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:33785
I20260927 13:55:04.821484 10688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:33785 every 8 connection(s)
I20260927 13:55:04.822795 10622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:04.826594 10689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:04.832850 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 10622
I20260927 13:55:04.833050 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:55:04.833633 21013 external_mini_cluster.cc:1428] Setting key f49e4cbe3bead42a044c4c90d730ca07
I20260927 13:55:04.838481 10689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b: Bootstrap starting.
I20260927 13:55:04.842522 10689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:04.844991 10689 log.cc:824] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:04.849182 10689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b: No bootstrap required, opened a new log
Sep 27 13:55:04 dist-test-slave-q25r krb5kdc[10609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:55:04.858572 10689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f182d9fafb6407c9bda807e3b629a3b" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33785 } }
I20260927 13:55:04.859171 10689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:04.859275 10689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f182d9fafb6407c9bda807e3b629a3b, State: Initialized, Role: FOLLOWER
I20260927 13:55:04.859915 10689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f182d9fafb6407c9bda807e3b629a3b" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33785 } }
I20260927 13:55:04.860275 10689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:55:04.860296 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:04.835582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60029 (local address 127.20.133.126:33785)
0927 13:55:04.836441 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:04.836453 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:04.836503 (+ 50us) server_negotiation.cc:407] Connection header received
0927 13:55:04.838193 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:04.838217 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:04.838640 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:04.839009 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:04.840838 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:04.842478 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:04.845270 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:04.846349 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:04.846611 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:04.851803 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:04.851814 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:04.851832 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:04.851910 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:04.855289 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:04.856417 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:04.856425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:04.856433 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:04.856706 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:04.857183 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:04.857188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:04.857191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:04.857866 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:04.858096 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:04.858624 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 13:55:04.859031 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":364,"threads_started":1}
I20260927 13:55:04.860798 10689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:55:04.861081 10689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:04.864096 10689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f182d9fafb6407c9bda807e3b629a3b" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33785 } }
I20260927 13:55:04.864708 10689 leader_election.cc:302] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f182d9fafb6407c9bda807e3b629a3b; no voters:
I20260927 13:55:04.865108 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:55:04.865370 10689 leader_election.cc:288] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:55:04.865471 10694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:04.866544 10694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [term 1 LEADER]: Becoming Leader. State: Replica: 2f182d9fafb6407c9bda807e3b629a3b, State: Running, Role: LEADER
I20260927 13:55:04.867089 10694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f182d9fafb6407c9bda807e3b629a3b" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33785 } }
I20260927 13:55:04.867599 10689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:04.878268 10696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f182d9fafb6407c9bda807e3b629a3b. Latest consensus state: current_term: 1 leader_uuid: "2f182d9fafb6407c9bda807e3b629a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f182d9fafb6407c9bda807e3b629a3b" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33785 } } }
I20260927 13:55:04.878746 10696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:04.880632 10695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f182d9fafb6407c9bda807e3b629a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f182d9fafb6407c9bda807e3b629a3b" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 33785 } } }
I20260927 13:55:04.881501 10695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:04.883562 10705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:04.889577 10705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:55:04.901693 10716 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 13:55:04.904023 10705 catalog_manager.cc:1347] Generated new cluster ID: 610b1ab6c805485da5907e8d6cc39ff7
I20260927 13:55:04.904224 10705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:04.911118 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:04.884364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39781 (local address 127.20.133.126:33785)
0927 13:55:04.884543 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:04.884550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:04.894936 (+ 10386us) server_negotiation.cc:407] Connection header received
0927 13:55:04.895128 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:04.895135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:04.895240 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:04.895386 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:04.898055 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:04.899370 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:04.901490 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:04.902196 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:04.902340 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:04.905440 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:04.905445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:04.905449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:04.905493 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:04.908202 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:04.908969 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:04.908974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:04.908976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:04.909127 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:04.909534 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:04.909537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:04.909540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:04.909898 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:04.910147 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:04.910498 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 13:55:04.910735 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:55:04.922665 10705 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:04.924232 10705 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:04.936693 10705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f182d9fafb6407c9bda807e3b629a3b: Generated new TSK 0
I20260927 13:55:04.937795 10705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:55:04.951853 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10622
I20260927 13:55:04.953078 10684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:05.046135 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 10622
2026-09-27T13:55:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:05 dist-test-slave-q25r krb5kdc[10721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:05 dist-test-slave-q25r krb5kdc[10721](info): set up 2 sockets
Sep 27 13:55:05 dist-test-slave-q25r krb5kdc[10721](info): commencing operation
krb5kdc: starting...
W20260927 13:55:06.093194 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.010s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:55:06 dist-test-slave-q25r krb5kdc[10721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:55:06.397790 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41723
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:06.732318 10734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:06.732725 10734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:06.732841 10734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:06.740881 10734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:06.741009 10734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:06.741063 10734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:06.741149 10734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:06.741214 10734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:06.751353 10734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:06.752743 10734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:06.754041 10734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:06.766340 10742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:06.766470 10740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:06.766350 10739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:06.768513 10734 server_base.cc:1034] running on GCE node
I20260927 13:55:06.769311 10734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:06.770741 10734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:06.771970 10734 hybrid_clock.cc:648] HybridClock initialized: now 1790517306771911 us; error 55 us; skew 500 ppm
I20260927 13:55:06.772360 10734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:06 dist-test-slave-q25r krb5kdc[10721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517306, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:06.775568 10734 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:55:06.777277 10734 webserver.cc:460] Webserver started at http://127.20.133.126:42963/ using document root <none> and password file <none>
I20260927 13:55:06.777951 10734 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:06.778067 10734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:06.778355 10734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:06.781314 10734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "e1870a8a6a2c446ebceb52686e11c627"
format_stamp: "Formatted at 2026-09-27 13:55:06 on dist-test-slave-q25r"
server_key: "dca876ba8c5cbd85e8c85f2f8d9a3b63"
server_key_iv: "d50e9a1818103ad61125021f8ae39c86"
server_key_version: "encryptionkey@0"
I20260927 13:55:06.782029 10734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "e1870a8a6a2c446ebceb52686e11c627"
format_stamp: "Formatted at 2026-09-27 13:55:06 on dist-test-slave-q25r"
server_key: "dca876ba8c5cbd85e8c85f2f8d9a3b63"
server_key_iv: "d50e9a1818103ad61125021f8ae39c86"
server_key_version: "encryptionkey@0"
I20260927 13:55:06.787568 10734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 13:55:06.791415 10749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:06.793103 10734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:55:06.793294 10734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "e1870a8a6a2c446ebceb52686e11c627"
format_stamp: "Formatted at 2026-09-27 13:55:06 on dist-test-slave-q25r"
server_key: "dca876ba8c5cbd85e8c85f2f8d9a3b63"
server_key_iv: "d50e9a1818103ad61125021f8ae39c86"
server_key_version: "encryptionkey@0"
I20260927 13:55:06.793484 10734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:06.814893 10734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:06.819123 10734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:06.819393 10734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:06.839004 10734 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:41723
I20260927 13:55:06.839079 10800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:41723 every 8 connection(s)
I20260927 13:55:06.840391 10734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:06.845283 10801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:06.848912 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 10734
I20260927 13:55:06.849138 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:55:06.849792 21013 external_mini_cluster.cc:1428] Setting key f6825c90a67697afc2e27505a7b01149
I20260927 13:55:06.859889 10801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627: Bootstrap starting.
I20260927 13:55:06.863832 10801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:06.865098 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:06.852545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43461 (local address 127.20.133.126:41723)
0927 13:55:06.853654 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:06.853670 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:06.853722 (+ 52us) server_negotiation.cc:407] Connection header received
0927 13:55:06.855756 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:06.855784 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:06.856315 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:06.856796 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:06.857425 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:06.858950 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:06.860293 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:06.861064 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:06.861249 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:06.862545 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:06.862554 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:06.862574 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:55:06.863155 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:06.863254 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:55:06.863282 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:06.863387 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:06.863608 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 13:55:06.863891 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":475,"threads_started":1}
I20260927 13:55:06.865180 10801 log.cc:824] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:06.868590 10801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627: No bootstrap required, opened a new log
I20260927 13:55:06.868993 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:55:06.877926 10801 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1870a8a6a2c446ebceb52686e11c627" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41723 } }
I20260927 13:55:06.878551 10801 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:06.878669 10801 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1870a8a6a2c446ebceb52686e11c627, State: Initialized, Role: FOLLOWER
I20260927 13:55:06.879417 10801 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1870a8a6a2c446ebceb52686e11c627" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41723 } }
I20260927 13:55:06.879702 10801 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:55:06.879873 10801 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:55:06.880067 10801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:06.880249 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:06.873413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44307 (local address 127.20.133.126:41723)
0927 13:55:06.873636 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:06.873643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:06.873883 (+ 240us) server_negotiation.cc:407] Connection header received
0927 13:55:06.874055 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:06.874061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:06.874158 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:06.874311 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:06.874891 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:06.876054 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:06.877516 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:06.878159 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:06.878301 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:06.879393 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:06.879400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:06.879409 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:55:06.879436 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:06.879486 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:55:06.879509 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:06.879602 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:06.879786 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 13:55:06.879897 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:55:06.882959 10801 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1870a8a6a2c446ebceb52686e11c627" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41723 } }
I20260927 13:55:06.883567 10801 leader_election.cc:302] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1870a8a6a2c446ebceb52686e11c627; no voters:
I20260927 13:55:06.884254 10801 leader_election.cc:288] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:55:06.884582 10811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:06.886194 10811 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [term 1 LEADER]: Becoming Leader. State: Replica: e1870a8a6a2c446ebceb52686e11c627, State: Running, Role: LEADER
I20260927 13:55:06.886969 10811 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1870a8a6a2c446ebceb52686e11c627" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41723 } }
I20260927 13:55:06.888068 10801 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:06.890070 10813 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1870a8a6a2c446ebceb52686e11c627. Latest consensus state: current_term: 1 leader_uuid: "e1870a8a6a2c446ebceb52686e11c627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1870a8a6a2c446ebceb52686e11c627" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41723 } } }
I20260927 13:55:06.890568 10813 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:06.895831 10812 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1870a8a6a2c446ebceb52686e11c627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1870a8a6a2c446ebceb52686e11c627" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 41723 } } }
I20260927 13:55:06.896467 10812 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:06.896857 10817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:06.903595 10817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:55:06.914842 10817 catalog_manager.cc:1347] Generated new cluster ID: 5305b1cb4ec64e6b9ba0b3fc546f2929
I20260927 13:55:06.914987 10817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:06.953752 10817 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:06.955190 10817 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:06.962617 10817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1870a8a6a2c446ebceb52686e11c627: Generated new TSK 0
I20260927 13:55:06.963526 10817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:55:06.979671 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10734
W20260927 13:55:06.981202 10796 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43461
W20260927 13:55:06.981986 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10734: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:55:06.982178 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 10734
2026-09-27T13:55:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:07 dist-test-slave-q25r krb5kdc[10832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:07 dist-test-slave-q25r krb5kdc[10832](info): set up 2 sockets
Sep 27 13:55:07 dist-test-slave-q25r krb5kdc[10832](info): commencing operation
krb5kdc: starting...
W20260927 13:55:08.028858 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:55:08 dist-test-slave-q25r krb5kdc[10832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:55:08.304036 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:35333
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:38403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:35333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:08.639532 10845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:08.639863 10845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:08.639957 10845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:08.647464 10845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:08.647603 10845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:08.647650 10845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:08.647711 10845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:08.647758 10845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:08.657796 10845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:38403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:35333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:35333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:08.659145 10845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:08.660394 10845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:08.672310 10850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:08.672407 10851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:08.672498 10853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:08.673127 10845 server_base.cc:1034] running on GCE node
I20260927 13:55:08.673969 10845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:08.675544 10845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:08.676798 10845 hybrid_clock.cc:648] HybridClock initialized: now 1790517308676736 us; error 49 us; skew 500 ppm
I20260927 13:55:08.677184 10845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:08 dist-test-slave-q25r krb5kdc[10832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517308, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:08.685402 10845 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:55:08.687098 10845 webserver.cc:460] Webserver started at http://127.20.133.126:34203/ using document root <none> and password file <none>
I20260927 13:55:08.687824 10845 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:08.687953 10845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:08.688261 10845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:08.691391 10845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "2119509087b74aa58c84b5614a93b7fb"
format_stamp: "Formatted at 2026-09-27 13:55:08 on dist-test-slave-q25r"
server_key: "f63e6aaa8432c8126d6dee7f1e5e29ef"
server_key_iv: "7e1f0ed6b7d2e82e1b4068ec75ef09ff"
server_key_version: "encryptionkey@0"
I20260927 13:55:08.692175 10845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "2119509087b74aa58c84b5614a93b7fb"
format_stamp: "Formatted at 2026-09-27 13:55:08 on dist-test-slave-q25r"
server_key: "f63e6aaa8432c8126d6dee7f1e5e29ef"
server_key_iv: "7e1f0ed6b7d2e82e1b4068ec75ef09ff"
server_key_version: "encryptionkey@0"
I20260927 13:55:08.697561 10845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:55:08.701264 10860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:08.702761 10845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:55:08.702948 10845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "2119509087b74aa58c84b5614a93b7fb"
format_stamp: "Formatted at 2026-09-27 13:55:08 on dist-test-slave-q25r"
server_key: "f63e6aaa8432c8126d6dee7f1e5e29ef"
server_key_iv: "7e1f0ed6b7d2e82e1b4068ec75ef09ff"
server_key_version: "encryptionkey@0"
I20260927 13:55:08.703157 10845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:08.709219 10845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:08.713653 10845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:08.713964 10845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:08.731863 10845 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:35333
I20260927 13:55:08.731892 10911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:35333 every 8 connection(s)
I20260927 13:55:08.733232 10845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:08.737020 10912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:08.740046 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 10845
I20260927 13:55:08.740200 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:55:08.740660 21013 external_mini_cluster.cc:1428] Setting key dc144080ae18e2384747c455347403c5
I20260927 13:55:08.749199 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:08.742762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54691 (local address 127.20.133.126:35333)
0927 13:55:08.743604 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:08.743617 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:08.743661 (+ 44us) server_negotiation.cc:407] Connection header received
0927 13:55:08.745539 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:08.745563 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:08.745943 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:08.746305 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:08.747507 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:08.747514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:08.747531 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:55:08.747557 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:08.747621 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:55:08.747642 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:08.747821 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:08.747962 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 13:55:08.748145 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":380,"threads_started":1}
I20260927 13:55:08.750138 10912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb: Bootstrap starting.
I20260927 13:55:08.753203 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:55:08.753795 10912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:08.754879 10912 log.cc:824] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:08.758373 10912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb: No bootstrap required, opened a new log
I20260927 13:55:08.760229 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:08.757872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46645 (local address 127.20.133.126:35333)
0927 13:55:08.758034 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:08.758041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:08.758407 (+ 366us) server_negotiation.cc:407] Connection header received
0927 13:55:08.758581 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:08.758587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:08.758652 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:08.758770 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:08.759556 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:08.759562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:08.759567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:55:08.759587 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:08.759644 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:55:08.759666 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:08.759791 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:08.759840 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 13:55:08.759900 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 13:55:08.766754 10912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119509087b74aa58c84b5614a93b7fb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35333 } }
I20260927 13:55:08.767165 10912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:55:08.767241 10912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2119509087b74aa58c84b5614a93b7fb, State: Initialized, Role: FOLLOWER
I20260927 13:55:08.767793 10912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119509087b74aa58c84b5614a93b7fb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35333 } }
I20260927 13:55:08.768004 10912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:55:08.768129 10912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:55:08.768260 10912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:08.771018 10912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119509087b74aa58c84b5614a93b7fb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35333 } }
I20260927 13:55:08.771567 10912 leader_election.cc:302] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2119509087b74aa58c84b5614a93b7fb; no voters:
I20260927 13:55:08.772228 10912 leader_election.cc:288] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:55:08.772408 10922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:08.773029 10922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [term 1 LEADER]: Becoming Leader. State: Replica: 2119509087b74aa58c84b5614a93b7fb, State: Running, Role: LEADER
I20260927 13:55:08.773829 10922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119509087b74aa58c84b5614a93b7fb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35333 } }
I20260927 13:55:08.774544 10912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:08.776824 10924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2119509087b74aa58c84b5614a93b7fb. Latest consensus state: current_term: 1 leader_uuid: "2119509087b74aa58c84b5614a93b7fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119509087b74aa58c84b5614a93b7fb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35333 } } }
I20260927 13:55:08.777060 10923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2119509087b74aa58c84b5614a93b7fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2119509087b74aa58c84b5614a93b7fb" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 35333 } } }
I20260927 13:55:08.777309 10924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:08.777398 10923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:08.778685 10931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:08.784195 10931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:55:08.793538 10931 catalog_manager.cc:1347] Generated new cluster ID: 35c848514bfb4b8fae22a7d1164b596a
I20260927 13:55:08.793669 10931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:08.803627 10931 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:08.804095 10931 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:08.818293 10931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2119509087b74aa58c84b5614a93b7fb: Generated new TSK 0
I20260927 13:55:08.819526 10931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:55:08.835549 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10845
W20260927 13:55:08.836934 10907 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54691
W20260927 13:55:08.837690 21013 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10845: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:55:08.837821 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 10845
2026-09-27T13:55:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1853 ms)
[----------] 18 tests from AuthTokenIssuingTest (35350 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:08 dist-test-slave-q25r krb5kdc[10943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:08 dist-test-slave-q25r krb5kdc[10943](info): set up 2 sockets
Sep 27 13:55:08 dist-test-slave-q25r krb5kdc[10943](info): commencing operation
krb5kdc: starting...
W20260927 13:55:09.864995 21013 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:55:10 dist-test-slave-q25r krb5kdc[10943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:55:10.168737 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:40305
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:40305,127.20.133.125:45065,127.20.133.124:35997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:10.512764 10956 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:55:10.513118 10956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:10.513217 10956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:10.513325 10956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:10.520735 10956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:10.520831 10956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:10.520891 10956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:10.520959 10956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:10.521046 10956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:10.531098 10956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:40305,127.20.133.125:45065,127.20.133.124:35997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:40305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:10.532444 10956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:10.533767 10956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:10.545344 10961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:10.545274 10962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:10.545306 10964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:10.546308 10956 server_base.cc:1034] running on GCE node
I20260927 13:55:10.547073 10956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:10.548599 10956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:10.549822 10956 hybrid_clock.cc:648] HybridClock initialized: now 1790517310549754 us; error 49 us; skew 500 ppm
I20260927 13:55:10.550159 10956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:10 dist-test-slave-q25r krb5kdc[10943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517310, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:10.554571 10956 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:55:10.556260 10956 webserver.cc:460] Webserver started at http://127.20.133.126:42121/ using document root <none> and password file <none>
I20260927 13:55:10.557101 10956 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:10.557214 10956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:10.557587 10956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:10.560958 10956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "a2d6f4f57f0347eba34de167c69433fe"
format_stamp: "Formatted at 2026-09-27 13:55:10 on dist-test-slave-q25r"
server_key: "c53bc5a5e2f57c6d63e1f2e454c14984"
server_key_iv: "13085c3deef259718a7623f23bae4dba"
server_key_version: "encryptionkey@0"
I20260927 13:55:10.561828 10956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "a2d6f4f57f0347eba34de167c69433fe"
format_stamp: "Formatted at 2026-09-27 13:55:10 on dist-test-slave-q25r"
server_key: "c53bc5a5e2f57c6d63e1f2e454c14984"
server_key_iv: "13085c3deef259718a7623f23bae4dba"
server_key_version: "encryptionkey@0"
I20260927 13:55:10.567605 10956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:55:10.571448 10971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:10.573043 10956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:55:10.573233 10956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "a2d6f4f57f0347eba34de167c69433fe"
format_stamp: "Formatted at 2026-09-27 13:55:10 on dist-test-slave-q25r"
server_key: "c53bc5a5e2f57c6d63e1f2e454c14984"
server_key_iv: "13085c3deef259718a7623f23bae4dba"
server_key_version: "encryptionkey@0"
I20260927 13:55:10.573467 10956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:10.597018 10956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:10.601415 10956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:10.601689 10956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:10.620187 10956 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:40305
I20260927 13:55:10.620239 11022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:40305 every 8 connection(s)
I20260927 13:55:10.621588 10956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:10.625375 11023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:10.628929 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 10956
I20260927 13:55:10.629153 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:55:10.629942 21013 external_mini_cluster.cc:1428] Setting key ef11ef8fc8df564749cbd8ce7eeb63ae
I20260927 13:55:10.634037 11023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:55:10 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517310, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.133.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.125@KRBTEST.COM" created.
I20260927 13:55:10.660771 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:10.638100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.20.133.126:40305)
0927 13:55:10.639831 (+ 1731us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:10.639842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:10.639891 (+ 49us) server_negotiation.cc:407] Connection header received
0927 13:55:10.641535 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:10.641541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:10.641789 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:10.641972 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:10.643511 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:10.644812 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:10.646785 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:10.647395 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:10.647556 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:10.651771 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:10.651779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:10.651794 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:10.651835 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:10.655228 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:10.656016 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:10.656023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:10.656025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:10.656183 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:10.656613 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:10.656618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:10.656620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:10.657149 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:10.657682 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:10.658681 (+ 999us) server_negotiation.cc:299] Negotiation successful
0927 13:55:10.659155 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1518,"thread_start_us":1291,"threads_started":1}
I20260927 13:55:10.660800 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:10.638100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:40305 (local address 127.0.0.1:46076)
0927 13:55:10.638992 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:10.639044 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:10.641132 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:10.642007 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:10.642015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:10.642786 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:10.643270 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:10.643295 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:10.644977 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:10.644982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:10.646575 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:10.646595 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:10.647669 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:10.647677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:10.647848 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:10.648634 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:10.648668 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:10.651493 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:10.655426 (+ 3933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:10.655439 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:10.655452 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:10.655871 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:10.656300 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:10.656305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:10.656309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:10.656507 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:10.657588 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:10.657606 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:10.658059 (+ 453us) client_negotiation.cc:769] Sending connection context
0927 13:55:10.658352 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 13:55:10.659334 (+ 982us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":168,"thread_start_us":393,"threads_started":1}
I20260927 13:55:10.664005 11023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:55:10.665290 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:10.664844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52020)
0927 13:55:10.665050 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:10.665164 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 13:55:10.665587 10975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
I20260927 13:55:10.667171 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:10.666764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52030)
0927 13:55:10.666952 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:10.667031 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 13:55:10.667375 11023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:45065: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab.
Entry for principal kudu/127.20.133.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab.
I20260927 13:55:10.689746 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.125:45065
--webserver_interface=127.20.133.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab
--principal=kudu/127.20.133.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:40305,127.20.133.125:45065,127.20.133.124:35997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:55:10.716727 11023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } attempt: 1
I20260927 13:55:10.717919 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:10.717450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52036)
0927 13:55:10.717660 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:10.717784 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 13:55:10.719030 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:10.718657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52052)
0927 13:55:10.718812 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:10.718890 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 13:55:10.719228 11023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:45065: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
I20260927 13:55:10.787510 11023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } attempt: 2
I20260927 13:55:10.788712 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:10.788176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52058)
0927 13:55:10.788479 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:10.788569 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 13:55:10.789908 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:10.789493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52074)
0927 13:55:10.789680 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:10.789771 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260927 13:55:10.790138 11023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:45065: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
I20260927 13:55:10.881393 11023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } attempt: 3
I20260927 13:55:10.882589 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:10.882142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52082)
0927 13:55:10.882342 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:10.882434 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 13:55:10.883704 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:10.883308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52094)
0927 13:55:10.883455 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:10.883555 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 13:55:10.883893 11023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:45065: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
I20260927 13:55:11.027148 11023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } attempt: 4
W20260927 13:55:11.027169 11030 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:55:11.027509 11030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:11.027621 11030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:11.027750 11030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 13:55:11.028472 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.027886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52110)
0927 13:55:11.028191 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.028284 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260927 13:55:11.029498 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.029162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52116)
0927 13:55:11.029294 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.029376 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 13:55:11.029685 11023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:45065: Network error: Client connection negotiation failed: client connection to 127.20.133.125:45065: connect: Connection refused (error 111)
W20260927 13:55:11.035741 11030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:11.035836 11030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:11.035893 11030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:11.035943 11030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:11.036005 11030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:11.046460 11030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:40305,127.20.133.125:45065,127.20.133.124:35997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab
--principal=kudu/127.20.133.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.125:45065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.133.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:11.047744 11030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:11.049072 11030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:11.060698 11035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:11.060772 11038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:11.060731 11036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:11.061676 11030 server_base.cc:1034] running on GCE node
I20260927 13:55:11.062470 11030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:11.064057 11030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:11.065299 11030 hybrid_clock.cc:648] HybridClock initialized: now 1790517311065209 us; error 63 us; skew 500 ppm
I20260927 13:55:11.065680 11030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517311, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:11.069036 11030 init.cc:377] Logged in from keytab as kudu/127.20.133.125@KRBTEST.COM (short username kudu)
I20260927 13:55:11.070688 11030 webserver.cc:460] Webserver started at http://127.20.133.125:46233/ using document root <none> and password file <none>
I20260927 13:55:11.071431 11030 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:11.071533 11030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:11.071808 11030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:11.074945 11030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/data/instance:
uuid: "04255dbdd12e4eb8ab9accbf293658d7"
format_stamp: "Formatted at 2026-09-27 13:55:11 on dist-test-slave-q25r"
server_key: "cab0e2bedd57f82e207e48fea793ebc7"
server_key_iv: "129fd2d538bd99692681311efcaa73ad"
server_key_version: "encryptionkey@0"
I20260927 13:55:11.075716 11030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/wal/instance:
uuid: "04255dbdd12e4eb8ab9accbf293658d7"
format_stamp: "Formatted at 2026-09-27 13:55:11 on dist-test-slave-q25r"
server_key: "cab0e2bedd57f82e207e48fea793ebc7"
server_key_iv: "129fd2d538bd99692681311efcaa73ad"
server_key_version: "encryptionkey@0"
I20260927 13:55:11.080947 11030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 13:55:11.084297 11045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:11.085808 11030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:55:11.085988 11030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/wal
uuid: "04255dbdd12e4eb8ab9accbf293658d7"
format_stamp: "Formatted at 2026-09-27 13:55:11 on dist-test-slave-q25r"
server_key: "cab0e2bedd57f82e207e48fea793ebc7"
server_key_iv: "129fd2d538bd99692681311efcaa73ad"
server_key_version: "encryptionkey@0"
I20260927 13:55:11.086205 11030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:11.119720 11030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:11.124063 11030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:11.124327 11030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:11.143131 11030 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.125:45065
I20260927 13:55:11.143155 11096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.125:45065 every 8 connection(s)
I20260927 13:55:11.144585 11030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
I20260927 13:55:11.145848 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 11030
I20260927 13:55:11.146039 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-1/wal/instance
I20260927 13:55:11.146580 21013 external_mini_cluster.cc:1428] Setting key e09ac894f77dd2040a5462d48db9c1ed
I20260927 13:55:11.149261 11097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:11.157887 11097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517311, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.133.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.124@KRBTEST.COM" created.
I20260927 13:55:11.179306 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.160879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.20.133.126:40305)
0927 13:55:11.161067 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.161072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.162364 (+ 1292us) server_negotiation.cc:407] Connection header received
0927 13:55:11.164441 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.164448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.164546 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.164741 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.166417 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.167109 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.168730 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.169267 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.169381 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.173015 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.173021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.173025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.173071 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.176166 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.176963 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.176969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.176972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.177116 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.177594 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.177599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.177602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.177942 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.178084 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.178786 (+ 702us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.178998 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:55:11.180192 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.161320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:40305 (local address 127.0.0.1:46090)
0927 13:55:11.162172 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.162214 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:11.164104 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:11.164992 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:11.165001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:11.165602 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:11.166219 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.166237 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.167273 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.167277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.168576 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.168590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.169496 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.169503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.169740 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.170486 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:11.170511 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:11.172777 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:11.176356 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.176367 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.176386 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.176845 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.177220 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.177224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.177226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.177474 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.178048 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:11.178054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:11.178443 (+ 389us) client_negotiation.cc:769] Sending connection context
0927 13:55:11.178671 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 13:55:11.179075 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":352,"threads_started":1}
I20260927 13:55:11.181965 11097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517311, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
I20260927 13:55:11.197619 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.182771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52132)
0927 13:55:11.182950 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.182970 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:11.183076 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:11.183973 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:11.183981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:11.184369 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:11.184515 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.184523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.186364 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.186368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.187200 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.187210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.188196 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.188200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.188365 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.188924 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:11.188940 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:11.191083 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:11.195026 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.195030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.195033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.195426 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.195901 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.195905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.195907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.196076 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.196858 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:11.196862 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:11.196988 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 13:55:11.197109 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 13:55:11.197342 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 13:55:11.197970 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.182908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.20.133.125:45065)
0927 13:55:11.183565 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.183571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.183595 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:55:11.183637 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.183643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.183821 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.183929 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.184687 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.186238 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.187326 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.188054 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.188168 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.191293 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.191304 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.191321 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.191356 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.194838 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.195602 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.195610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.195614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.195766 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.196188 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.196192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.196194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.196720 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.196922 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.197274 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.197537 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":369,"threads_started":1}
I20260927 13:55:11.200613 11097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:55:11.201720 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.201249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43222)
0927 13:55:11.201422 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.201562 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 13:55:11.202000 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
I20260927 13:55:11.203512 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.203152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43238)
0927 13:55:11.203319 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.203401 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 13:55:11.203723 11097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:35997: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab.
Entry for principal kudu/127.20.133.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab.
I20260927 13:55:11.211237 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.124:35997
--webserver_interface=127.20.133.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab
--principal=kudu/127.20.133.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:40305,127.20.133.125:45065,127.20.133.124:35997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:55:11.253095 11097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } attempt: 1
I20260927 13:55:11.254976 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.254174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43242)
0927 13:55:11.254523 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.254712 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260927 13:55:11.256623 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.255903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43252)
0927 13:55:11.256188 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.256399 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 13:55:11.256973 11097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:35997: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
I20260927 13:55:11.325275 11097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } attempt: 2
I20260927 13:55:11.326934 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.326254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43260)
0927 13:55:11.326555 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.326715 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 13:55:11.328212 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.327747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43264)
0927 13:55:11.327936 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.328051 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 13:55:11.328509 11097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:35997: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
I20260927 13:55:11.328961 11023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } attempt: 5
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517310, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
I20260927 13:55:11.345556 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.330002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.20.133.125:45065)
0927 13:55:11.330201 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.330207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.330235 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:55:11.330382 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.330388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.330506 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.330651 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.331698 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.332795 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.334200 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.334889 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.335028 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.339315 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.339322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.339326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.339378 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.342428 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.343490 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.343495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.343498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.343644 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.344058 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.344061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.344063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.344457 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.344585 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.345058 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.345275 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:55:11.345715 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.329766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52146)
0927 13:55:11.330031 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.330056 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:11.330204 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:11.330724 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:11.330728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:11.331228 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:11.331510 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.331521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.332979 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.332985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.333998 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.334016 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.335107 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.335113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.335338 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.336138 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:11.336160 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:11.339017 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:11.342655 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.342662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.342666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.343329 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.343759 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.343764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.343768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.343953 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.344576 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:11.344590 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:11.344793 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 13:55:11.344966 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 13:55:11.345337 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 13:55:11.347759 11023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:55:11.349303 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.348596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43272)
0927 13:55:11.348931 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.349066 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 13:55:11.350943 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.350487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43278)
0927 13:55:11.350683 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.350807 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260927 13:55:11.351289 11023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:35997: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
I20260927 13:55:11.402627 11023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } attempt: 1
I20260927 13:55:11.404101 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.403538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43284)
0927 13:55:11.403777 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.403886 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 13:55:11.405550 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.405123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43294)
0927 13:55:11.405333 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.405423 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 13:55:11.405810 11023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:35997: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
I20260927 13:55:11.419818 11097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } attempt: 3
I20260927 13:55:11.421602 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.420864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43308)
0927 13:55:11.421195 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.421347 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260927 13:55:11.423146 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.422510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43318)
0927 13:55:11.422834 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.422985 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
W20260927 13:55:11.423501 11097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:35997: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
I20260927 13:55:11.474086 11023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } attempt: 2
I20260927 13:55:11.475652 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.474977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43324)
0927 13:55:11.475321 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.475444 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 13:55:11.476891 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.476519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43326)
0927 13:55:11.476682 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.476761 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 13:55:11.477180 11023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:35997: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
W20260927 13:55:11.555150 11104 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:55:11.555490 11104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:11.555554 11104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:11.555641 11104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:11.563200 11104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:11.563336 11104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:11.563395 11104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:11.563437 11104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:11.563503 11104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:11.566804 11097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } attempt: 4
I20260927 13:55:11.568431 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.567686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43328)
0927 13:55:11.568012 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.568135 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 13:55:11.569634 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.569232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43344)
0927 13:55:11.569430 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.569515 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 13:55:11.569890 11097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:35997: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
I20260927 13:55:11.574339 11104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:40305,127.20.133.125:45065,127.20.133.124:35997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab
--principal=kudu/127.20.133.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.124:35997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.133.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:11.575692 11104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:11.577279 11104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 13:55:11.583473 11023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } attempt: 3
I20260927 13:55:11.584824 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.584187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43348)
0927 13:55:11.584489 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.584615 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 13:55:11.586058 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.585689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43364)
0927 13:55:11.585853 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.585944 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 13:55:11.586297 11023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:35997: Network error: Client connection negotiation failed: client connection to 127.20.133.124:35997: connect: Connection refused (error 111)
W20260927 13:55:11.590291 11109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:11.590920 11110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:11.590991 11112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:11.592590 11104 server_base.cc:1034] running on GCE node
I20260927 13:55:11.593701 11104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:11.595769 11104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:11.597036 11104 hybrid_clock.cc:648] HybridClock initialized: now 1790517311596952 us; error 54 us; skew 500 ppm
I20260927 13:55:11.597460 11104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517311, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:11.601447 11104 init.cc:377] Logged in from keytab as kudu/127.20.133.124@KRBTEST.COM (short username kudu)
I20260927 13:55:11.603147 11104 webserver.cc:460] Webserver started at http://127.20.133.124:46521/ using document root <none> and password file <none>
I20260927 13:55:11.603897 11104 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:11.603994 11104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:11.604287 11104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:11.607404 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/data/instance:
uuid: "d77c7e53f9fa43b680142537241375e3"
format_stamp: "Formatted at 2026-09-27 13:55:11 on dist-test-slave-q25r"
server_key: "77f47389b8bdf07e47b9461c00b27ed2"
server_key_iv: "f8ad449c1fda3dbb94d9eabf71ce23c2"
server_key_version: "encryptionkey@0"
I20260927 13:55:11.608198 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/wal/instance:
uuid: "d77c7e53f9fa43b680142537241375e3"
format_stamp: "Formatted at 2026-09-27 13:55:11 on dist-test-slave-q25r"
server_key: "77f47389b8bdf07e47b9461c00b27ed2"
server_key_iv: "f8ad449c1fda3dbb94d9eabf71ce23c2"
server_key_version: "encryptionkey@0"
I20260927 13:55:11.613888 11104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 13:55:11.618177 11119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:11.620309 11104 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260927 13:55:11.620595 11104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/wal
uuid: "d77c7e53f9fa43b680142537241375e3"
format_stamp: "Formatted at 2026-09-27 13:55:11 on dist-test-slave-q25r"
server_key: "77f47389b8bdf07e47b9461c00b27ed2"
server_key_iv: "f8ad449c1fda3dbb94d9eabf71ce23c2"
server_key_version: "encryptionkey@0"
I20260927 13:55:11.620901 11104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:11.648692 11104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:11.653062 11104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:11.653317 11104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:11.672995 11104 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.124:35997
I20260927 13:55:11.673054 11170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.124:35997 every 8 connection(s)
I20260927 13:55:11.674425 11104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
I20260927 13:55:11.678774 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 11104
I20260927 13:55:11.679023 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516948056776-21013-0/minicluster-data/master-2/wal/instance
I20260927 13:55:11.679679 21013 external_mini_cluster.cc:1428] Setting key 5dde59a39297da546d936c362a9854f8
I20260927 13:55:11.679761 11171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:11.690285 11171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517311, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:55:11.704372 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.681980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.20.133.124:35997)
0927 13:55:11.682963 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.682978 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.683030 (+ 52us) server_negotiation.cc:407] Connection header received
0927 13:55:11.685333 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.685361 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.685904 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.686331 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.687883 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.689279 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.690770 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.691572 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.691763 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.696544 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.696554 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.696569 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.696627 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.699722 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.700580 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.700586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.700594 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.700773 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.701168 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.701171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.701173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.701888 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.702229 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.702696 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.703152 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":437,"threads_started":1}
I20260927 13:55:11.711041 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.693527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:40305 (local address 127.0.0.1:46100)
0927 13:55:11.695017 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.695079 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:11.695344 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:11.695901 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:11.695908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:11.696883 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:11.697324 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.697337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.698583 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.698587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.699915 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.699930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.700801 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.700808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.700993 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.701690 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:11.701708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:11.704220 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:11.707952 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.707964 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.707972 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.708667 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.709097 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.709101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.709104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.709274 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.709962 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:11.709969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:11.710155 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 13:55:11.710323 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 13:55:11.710617 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192,"thread_start_us":1041,"threads_started":1}
I20260927 13:55:11.711102 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.693609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.20.133.126:40305)
0927 13:55:11.694350 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.694359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.695229 (+ 870us) server_negotiation.cc:407] Connection header received
0927 13:55:11.695527 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.695533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.695636 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.695787 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.697519 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.698409 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.700081 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.700591 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.700713 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.704480 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.704487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.704492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.704543 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.707781 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.708828 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.708832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.708834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.708984 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.709403 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.709408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.709411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.709801 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.709968 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.710467 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.710717 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":339,"threads_started":1}
I20260927 13:55:11.712931 11171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517311, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
I20260927 13:55:11.725694 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.708037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46110 (local address 127.20.133.126:40305)
0927 13:55:11.708805 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.708812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.708838 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:55:11.708905 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.708910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.708976 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.709086 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.710795 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.712160 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.713573 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.714099 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.714215 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.719295 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.719304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.719309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.719357 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.722494 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.723373 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.723379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.723382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.723538 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.723889 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.723893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.723896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.724287 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.724633 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.724954 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.725216 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":360,"threads_started":1}
I20260927 13:55:11.729362 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.713789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:45065 (local address 127.0.0.1:52150)
0927 13:55:11.714020 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.714043 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:11.714162 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:11.714567 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:11.714573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:11.715032 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:11.715214 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.715225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.716242 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.716248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.717594 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.717607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.718590 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.718597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.718811 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.719554 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:11.719575 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:11.721959 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:11.725980 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.725987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.725992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.726597 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.727255 (+ 658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.727260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.727263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.727475 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.728210 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:11.728215 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:11.728463 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 13:55:11.728608 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 13:55:11.728975 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 13:55:11.729436 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.714078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.20.133.125:45065)
0927 13:55:11.714258 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.714263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.714279 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:55:11.714338 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.714342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.714408 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.714489 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.715370 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.716091 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.717784 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.718396 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.718500 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.722209 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.722216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.722221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.722283 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.725766 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.726795 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.726801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.726804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.727073 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.727615 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.727620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.727623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.728062 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.728253 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.728730 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.729000 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:55:11.730654 11171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517311, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
I20260927 13:55:11.745244 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.731643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.20.133.124:35997)
0927 13:55:11.731841 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.731846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.731862 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:55:11.731908 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.731912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.731973 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.732056 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.733068 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.733981 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.735636 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.736071 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.736189 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.739922 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.739927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.739930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.739962 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.742669 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.743328 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.743334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.743336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.743442 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.743763 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.743768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.743771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.744061 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.744186 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.744597 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.744959 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":64}
I20260927 13:55:11.745216 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.731419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43392)
0927 13:55:11.731597 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.731618 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:11.731724 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:11.732121 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:11.732126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:11.732712 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:11.732921 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.732933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.734142 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.734146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.735438 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.735456 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.736283 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.736290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.736576 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.737228 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:11.737254 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:11.739724 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:11.742819 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.742824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.742829 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.743221 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.743539 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.743544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.743548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.743668 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.744171 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:11.744175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:11.744310 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 13:55:11.744473 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 13:55:11.744793 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 13:55:11.745949 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.728284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.20.133.125:45065)
0927 13:55:11.728932 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.728936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.728957 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:55:11.729005 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.729016 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.729082 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.729188 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.730666 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.731671 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.733452 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.734111 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.734214 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.739786 (+ 5572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.739793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.739796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.739847 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.742824 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.743573 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.743578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.743580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.743720 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.744102 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.744105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.744107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.744470 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.744747 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.745033 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.745291 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":274,"threads_started":1}
I20260927 13:55:11.751382 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:55:11.751821 11171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3: Bootstrap starting.
I20260927 13:55:11.756181 11171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:11.757419 11171 log.cc:824] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:11.755240 10998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a2d6f4f57f0347eba34de167c69433fe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46110
I20260927 13:55:11.760820 11171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3: No bootstrap required, opened a new log
I20260927 13:55:11.763566 11023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } attempt: 4
I20260927 13:55:11.770212 11171 raft_consensus.cc:348] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } }
I20260927 13:55:11.770740 11171 raft_consensus.cc:729] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77c7e53f9fa43b680142537241375e3, State: Initialized, Role: FOLLOWER
I20260927 13:55:11.771361 11171 consensus_queue.cc:260] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } }
I20260927 13:55:11.772651 11180 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } } }
I20260927 13:55:11.773166 11180 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517310, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
I20260927 13:55:11.774385 11171 sys_catalog.cc:564] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:11.782953 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.764371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43402)
0927 13:55:11.764733 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.764754 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:11.764899 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:11.765360 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:11.765365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:11.765952 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:11.766217 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.766229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.767802 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.767808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.769387 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.769402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.770476 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.770483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.770711 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.771589 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:11.771615 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:11.774872 (+ 3257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:11.779336 (+ 4464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.779343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.779347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.779981 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.780619 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.780625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.780628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.780831 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.781799 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:11.781805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:11.782016 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 13:55:11.782169 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 13:55:11.782522 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260927 13:55:11.784116 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.764537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.20.133.124:35997)
0927 13:55:11.764775 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.764781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.764808 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:55:11.765061 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.765067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.765161 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.765304 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.766368 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.767625 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.769549 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.770300 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.770429 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.775181 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.775190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.775196 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.775248 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.779151 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.780208 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.780215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.780220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.780476 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.781011 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.781018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.781021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.781550 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.781732 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.783336 (+ 1604us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.783603 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 13:55:11.790989 11023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe: Bootstrap starting.
I20260927 13:55:11.784101 11104 master_runner.cc:186] Error getting master registration for 127.20.133.126:40305: OK, instance_id { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" instance_seqno: 1790517310615023 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:55:11.795549 11104 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 13:55:11.795755 11023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:11.797266 11023 log.cc:824] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe: Log is configured to *not* fsync() on all Append() calls
W20260927 13:55:11.798563 11191 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:55:11.798722 11191 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:55:11.801376 11023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe: No bootstrap required, opened a new log
I20260927 13:55:11.809405 11023 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } }
I20260927 13:55:11.809859 11023 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d6f4f57f0347eba34de167c69433fe, State: Initialized, Role: FOLLOWER
I20260927 13:55:11.810385 11023 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } }
I20260927 13:55:11.811558 11194 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } } }
I20260927 13:55:11.812181 11194 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:11.812834 11023 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:11.812906 10998 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 13:55:11.813113 10998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:11.817179 10998 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } }
I20260927 13:55:11.819718 10998 leader_election.cc:288] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [CANDIDATE]: Term 1 election: Requested vote from peers 04255dbdd12e4eb8ab9accbf293658d7 (127.20.133.125:45065), d77c7e53f9fa43b680142537241375e3 (127.20.133.124:35997)
I20260927 13:55:11.820250 11146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2d6f4f57f0347eba34de167c69433fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d77c7e53f9fa43b680142537241375e3"
I20260927 13:55:11.820935 11146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:11.824610 11146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2d6f4f57f0347eba34de167c69433fe in term 1.
I20260927 13:55:11.825395 10975 leader_election.cc:302] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2d6f4f57f0347eba34de167c69433fe, d77c7e53f9fa43b680142537241375e3; no voters:
I20260927 13:55:11.826331 11194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:11.848618 11194 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [term 1 LEADER]: Becoming Leader. State: Replica: a2d6f4f57f0347eba34de167c69433fe, State: Running, Role: LEADER
I20260927 13:55:11.849421 11194 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } }
W20260927 13:55:11.826174 11205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:55:11.854930 10977 tablet.cc:2369] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:55:11.854809 11205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:55:11.858932 11208 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d6f4f57f0347eba34de167c69433fe. Latest consensus state: current_term: 1 leader_uuid: "a2d6f4f57f0347eba34de167c69433fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } } }
I20260927 13:55:11.859304 11208 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:11.860505 11213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:11.869127 11097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } attempt: 5
I20260927 13:55:11.870587 11213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 13:55:11 dist-test-slave-q25r krb5kdc[10943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517311, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
I20260927 13:55:11.827719 10956 master_runner.cc:186] Error getting master registration for 127.20.133.125:45065: OK, instance_id { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" instance_seqno: 1790517311137368 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:55:11.882083 10956 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 13:55:11.890506 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.870231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:35997 (local address 127.0.0.1:43410)
0927 13:55:11.870656 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:11.870679 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:11.870854 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:11.872031 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:11.872037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:11.872740 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:11.872987 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.873004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.874598 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.874604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.876261 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:11.876280 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.878038 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.878045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:11.878412 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.879218 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:11.879254 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:11.882571 (+ 3317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:11.887227 (+ 4656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.887234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.887239 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.887891 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.888453 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:11.888458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:11.888461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:11.888645 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:11.889375 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:11.889381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:11.889567 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 13:55:11.889719 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 13:55:11.890065 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295}
I20260927 13:55:11.891541 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.871093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.20.133.124:35997)
0927 13:55:11.871297 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.871304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.871326 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:55:11.871406 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.871411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.871501 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.871611 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.873187 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.874404 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.876505 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.877653 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.877781 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.882802 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.882810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.882815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.882864 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.886982 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.888066 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.888072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.888076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.888272 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.888786 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.888791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.888794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.889177 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.889313 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.890882 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.891155 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 13:55:11.905893 11125 tablet.cc:2369] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:55:11.910020 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.863802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.20.133.126:40305)
0927 13:55:11.864085 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.864092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.864586 (+ 494us) server_negotiation.cc:407] Connection header received
0927 13:55:11.865090 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.865096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.865192 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.865277 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.884273 (+ 18996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.885366 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.888980 (+ 3614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.889647 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.889740 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.899941 (+ 10201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.899949 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.899954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.900011 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.904965 (+ 4954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.906752 (+ 1787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.906759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.906763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.906942 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.907706 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.907712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.907716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.908133 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.908443 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.909395 (+ 952us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.909648 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 13:55:11.914095 11146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [term 1 FOLLOWER]: Refusing update from remote peer a2d6f4f57f0347eba34de167c69433fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:11.915261 11194 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:11.915437 11097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7: Bootstrap starting.
I20260927 13:55:11.920560 11097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:11.923158 11097 log.cc:824] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:11.934600 11097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7: No bootstrap required, opened a new log
I20260927 13:55:11.948216 11097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } }
I20260927 13:55:11.948879 11097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04255dbdd12e4eb8ab9accbf293658d7, State: Initialized, Role: FOLLOWER
I20260927 13:55:11.949581 11097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } }
I20260927 13:55:11.954294 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.865010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.20.133.125:45065)
0927 13:55:11.865230 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.865237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.887136 (+ 21899us) server_negotiation.cc:407] Connection header received
0927 13:55:11.903385 (+ 16249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.903393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.903530 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.903625 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.921479 (+ 17854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.922673 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.928987 (+ 6314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.929786 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.929904 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.938972 (+ 9068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.938985 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.938990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.939050 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.943115 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.949107 (+ 5992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.949113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.949117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.949303 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.950944 (+ 1641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.950949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.950953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.951405 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.951660 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.953526 (+ 1866us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.953866 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:55:11.958257 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:11.887824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.20.133.124:35997)
0927 13:55:11.891829 (+ 4005us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:11.891835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:11.902762 (+ 10927us) server_negotiation.cc:407] Connection header received
0927 13:55:11.906872 (+ 4110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:11.906879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:11.906979 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:11.907057 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:11.925641 (+ 18584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.926854 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.931182 (+ 4328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:11.931972 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:11.932077 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:11.944289 (+ 12212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:11.944299 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:11.944304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:11.944387 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:11.948083 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.950176 (+ 2093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.950183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.950190 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.950365 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:11.952184 (+ 1819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:11.952190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:11.952193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:11.952691 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:11.952933 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:11.957371 (+ 4438us) server_negotiation.cc:299] Negotiation successful
0927 13:55:11.957721 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3851,"thread_start_us":20809,"threads_started":1}
I20260927 13:55:11.965124 11180 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d6f4f57f0347eba34de167c69433fe. Latest consensus state: current_term: 1 leader_uuid: "a2d6f4f57f0347eba34de167c69433fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } } }
I20260927 13:55:11.965540 11180 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:11.969127 11221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } } }
I20260927 13:55:11.969772 11221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:11.970894 11097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:11.971386 11071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:11.971913 11072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2d6f4f57f0347eba34de167c69433fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "04255dbdd12e4eb8ab9accbf293658d7"
I20260927 13:55:11.976879 11071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [term 1 FOLLOWER]: Refusing update from remote peer a2d6f4f57f0347eba34de167c69433fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:11.980979 11072 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2d6f4f57f0347eba34de167c69433fe in term 1.
I20260927 13:55:11.983726 11208 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:12.000283 11180 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2d6f4f57f0347eba34de167c69433fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } } }
I20260927 13:55:12.000670 11180 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:12.011834 11208 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2d6f4f57f0347eba34de167c69433fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } } }
I20260927 13:55:12.012169 11208 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:12.013846 11227 mvcc.cc:205] Tried to move back new op lower bound from 7333958909587042304 to 7333958909344468992. Current Snapshot: MvccSnapshot[applied={T|T < 7333958909587042304}]
I20260927 13:55:12.023067 11229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2d6f4f57f0347eba34de167c69433fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } } }
I20260927 13:55:12.023592 11229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:12.024227 11213 catalog_manager.cc:1347] Generated new cluster ID: 31cb612478274f4189ee6cb0dab40f88
I20260927 13:55:12.024360 11213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:12.025390 11221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d6f4f57f0347eba34de167c69433fe. Latest consensus state: current_term: 1 leader_uuid: "a2d6f4f57f0347eba34de167c69433fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } } }
I20260927 13:55:12.029477 11221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:12.078531 11208 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2d6f4f57f0347eba34de167c69433fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d6f4f57f0347eba34de167c69433fe" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 40305 } } peers { permanent_uuid: "04255dbdd12e4eb8ab9accbf293658d7" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 45065 } } peers { permanent_uuid: "d77c7e53f9fa43b680142537241375e3" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 35997 } } }
I20260927 13:55:12.078882 11208 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:12.089910 11213 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:12.092264 11213 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:12.111821 11213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2d6f4f57f0347eba34de167c69433fe: Generated new TSK 0
I20260927 13:55:12.113200 11213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:55:12.133543 11240 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260927 13:55:12.234611 11177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.223944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.20.133.126:40305)
0927 13:55:12.224311 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.224317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.230573 (+ 6256us) server_negotiation.cc:407] Connection header received
0927 13:55:12.232583 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.232591 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.232698 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.232796 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.234245 (+ 1449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46124: BlockingRecv error: recv got EOF from 127.0.0.1:46124 (error 108)
Metrics: {"server-negotiator.queue_time_us":249}
W20260927 13:55:12.237528 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.223736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.20.133.125:45065)
0927 13:55:12.224167 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.224172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.235374 (+ 11202us) server_negotiation.cc:407] Connection header received
0927 13:55:12.235692 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.235698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.235797 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.235874 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.237248 (+ 1374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52182: BlockingRecv error: recv got EOF from 127.0.0.1:52182 (error 108)
Metrics: {"server-negotiator.queue_time_us":279}
W20260927 13:55:12.239434 11174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.231475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.20.133.124:35997)
0927 13:55:12.231695 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.231701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.236995 (+ 5294us) server_negotiation.cc:407] Connection header received
0927 13:55:12.237691 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.237697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.237843 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.237923 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.239129 (+ 1206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43422: BlockingRecv error: recv got EOF from 127.0.0.1:43422 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260927 13:55:12.254894 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.252400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.20.133.125:45065)
0927 13:55:12.252653 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.252660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.253155 (+ 495us) server_negotiation.cc:407] Connection header received
0927 13:55:12.253448 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.253454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.253594 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.253721 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.254646 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52186: BlockingRecv error: recv got EOF from 127.0.0.1:52186 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 13:55:12.262477 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.259546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.20.133.125:45065)
0927 13:55:12.259727 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.259733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.260874 (+ 1141us) server_negotiation.cc:407] Connection header received
0927 13:55:12.261081 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.261087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.261196 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.261301 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.262199 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52192: BlockingRecv error: recv got EOF from 127.0.0.1:52192 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 13:55:12.279251 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.271330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.20.133.125:45065)
0927 13:55:12.271631 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.271637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.277155 (+ 5518us) server_negotiation.cc:407] Connection header received
0927 13:55:12.277491 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.277498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.277634 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.277754 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.278989 (+ 1235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52204: BlockingRecv error: recv got EOF from 127.0.0.1:52204 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
W20260927 13:55:12.298739 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.291750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.20.133.125:45065)
0927 13:55:12.291993 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.291999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.297044 (+ 5045us) server_negotiation.cc:407] Connection header received
0927 13:55:12.297263 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.297270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.297389 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.297514 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.298469 (+ 955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52206: BlockingRecv error: recv got EOF from 127.0.0.1:52206 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
W20260927 13:55:12.314625 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.312067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.20.133.125:45065)
0927 13:55:12.312281 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.312286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.312916 (+ 630us) server_negotiation.cc:407] Connection header received
0927 13:55:12.313232 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.313238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.313333 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.313452 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.314389 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52212: BlockingRecv error: recv got EOF from 127.0.0.1:52212 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 13:55:12.336674 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.334336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.20.133.125:45065)
0927 13:55:12.334540 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.334546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.335049 (+ 503us) server_negotiation.cc:407] Connection header received
0927 13:55:12.335279 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.335285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.335390 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.335505 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.336405 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52218: BlockingRecv error: recv got EOF from 127.0.0.1:52218 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 13:55:12.374847 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.372545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.20.133.125:45065)
0927 13:55:12.372756 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.372762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.373342 (+ 580us) server_negotiation.cc:407] Connection header received
0927 13:55:12.373552 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.373558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.373652 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.373758 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.374615 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52228: BlockingRecv error: recv got EOF from 127.0.0.1:52228 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 13:55:12.444056 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.442267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.20.133.125:45065)
0927 13:55:12.442475 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.442481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.442797 (+ 316us) server_negotiation.cc:407] Connection header received
0927 13:55:12.442930 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.442935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.443025 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.443121 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.443872 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52236: BlockingRecv error: recv got EOF from 127.0.0.1:52236 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 13:55:12.577790 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.575705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.20.133.125:45065)
0927 13:55:12.575916 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.575922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.576251 (+ 329us) server_negotiation.cc:407] Connection header received
0927 13:55:12.576435 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.576440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.576523 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.576616 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.577521 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52242: BlockingRecv error: recv got EOF from 127.0.0.1:52242 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 13:55:12.800750 11191 catalog_manager.cc:1259] Loaded cluster ID: 31cb612478274f4189ee6cb0dab40f88
I20260927 13:55:12.800868 11191 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3: loading cluster ID for follower catalog manager: success
I20260927 13:55:12.803400 11191 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3: acquiring CA information for follower catalog manager: success
I20260927 13:55:12.805384 11191 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d77c7e53f9fa43b680142537241375e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 13:55:12.840067 11178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:12.838189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.20.133.125:45065)
0927 13:55:12.838405 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:12.838412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:12.838530 (+ 118us) server_negotiation.cc:407] Connection header received
0927 13:55:12.838766 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:12.838771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:12.838848 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:12.838943 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:12.839776 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52256: BlockingRecv error: recv got EOF from 127.0.0.1:52256 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:55:13.138413 11240 catalog_manager.cc:1259] Loaded cluster ID: 31cb612478274f4189ee6cb0dab40f88
I20260927 13:55:13.138528 11240 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7: loading cluster ID for follower catalog manager: success
I20260927 13:55:13.141274 11240 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7: acquiring CA information for follower catalog manager: success
I20260927 13:55:13.143428 11240 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04255dbdd12e4eb8ab9accbf293658d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:55:13.365691 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:13.357296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.20.133.125:45065)
0927 13:55:13.358061 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:13.358072 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:13.358480 (+ 408us) server_negotiation.cc:407] Connection header received
0927 13:55:13.358711 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:13.358717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:13.358935 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:13.359116 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:13.359696 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:13.360980 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:13.362509 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:13.363263 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:13.363430 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:13.364016 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:13.364923 (+ 907us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:13.365088 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:13.365187 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 13:55:13.365291 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":329,"threads_started":1}
I20260927 13:55:13.382812 11312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:13.373681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.20.133.124:35997)
0927 13:55:13.376798 (+ 3117us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:13.376804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:13.376829 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:55:13.376960 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:13.376964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:13.377219 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:13.377463 (+ 244us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:13.378087 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:13.378961 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:13.380327 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:13.380781 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:13.380893 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:13.381430 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:13.382143 (+ 713us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:13.382280 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:13.382381 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 13:55:13.382475 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2973,"thread_start_us":2840,"threads_started":1}
I20260927 13:55:13.409312 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:13.400823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.20.133.125:45065)
0927 13:55:13.401072 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:13.401079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:13.401741 (+ 662us) server_negotiation.cc:407] Connection header received
0927 13:55:13.402044 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:13.402050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:13.402150 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:13.402222 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:13.403091 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:13.404508 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:13.406762 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:13.407537 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:13.407631 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:13.408258 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:13.408448 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:13.408512 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:13.408781 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 13:55:13.408884 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:55:13.414121 11312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:13.401991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.20.133.124:35997)
0927 13:55:13.402183 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:13.402190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:13.406073 (+ 3883us) server_negotiation.cc:407] Connection header received
0927 13:55:13.406488 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:13.406494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:13.406599 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:13.406666 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:13.407422 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:13.408515 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:13.411645 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:13.412298 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:13.412425 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:13.413131 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:13.413355 (+ 224us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:13.413493 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:13.413668 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 13:55:13.413808 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:55:13.418098 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:13.400035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.20.133.126:40305)
0927 13:55:13.410838 (+ 10803us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:13.410845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:13.410874 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:55:13.411023 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:13.411027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:13.411230 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:13.411420 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:13.412026 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:13.413111 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:13.414904 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:13.415579 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:13.415704 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:13.416680 (+ 976us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:13.417268 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:13.417434 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:13.417633 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 13:55:13.417756 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10639,"thread_start_us":10342,"threads_started":1}
I20260927 13:55:13.425388 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 10956
I20260927 13:55:13.426932 11018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:13.558606 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 10956
W20260927 13:55:13.576747 11123 connection.cc:535] server connection from 127.0.0.1:43402 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 13:55:13.577104 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 11030
I20260927 13:55:13.578562 11092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:13.690734 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 11030
I20260927 13:55:13.707350 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 11104
I20260927 13:55:13.708487 11166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:13.828747 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 11104
2026-09-27T13:55:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4997 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:55:13 dist-test-slave-q25r krb5kdc[11326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:55:13 dist-test-slave-q25r krb5kdc[11326](info): set up 2 sockets
Sep 27 13:55:13 dist-test-slave-q25r krb5kdc[11326](info): commencing operation
krb5kdc: starting...
W20260927 13:55:14.884791 21013 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.007s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/test-user.keytab.
Sep 27 13:55:15 dist-test-slave-q25r krb5kdc[11326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:55:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:55:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.133.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
Entry for principal kudu/127.20.133.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab.
I20260927 13:55:15.156901 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:44561
--webserver_interface=127.20.133.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:44561,127.20.133.125:35957,127.20.133.124:46607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:55:15.516808 11339 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:55:15.517179 11339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:15.517259 11339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:15.517347 11339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:15.524919 11339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:15.525029 11339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:15.525094 11339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:15.525152 11339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:15.525225 11339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:15.535276 11339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:44561,127.20.133.125:35957,127.20.133.124:46607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.126.keytab
--principal=kudu/127.20.133.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.126:44561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.133.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:15.536644 11339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:15.537919 11339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:15.551007 11345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:15.551085 11347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:15.551040 11344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:15.552165 11339 server_base.cc:1034] running on GCE node
I20260927 13:55:15.553131 11339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:15.554682 11339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:15.555948 11339 hybrid_clock.cc:648] HybridClock initialized: now 1790517315555856 us; error 68 us; skew 500 ppm
I20260927 13:55:15.556408 11339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:15 dist-test-slave-q25r krb5kdc[11326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517315, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:15.559836 11339 init.cc:377] Logged in from keytab as kudu/127.20.133.126@KRBTEST.COM (short username kudu)
I20260927 13:55:15.561546 11339 webserver.cc:460] Webserver started at http://127.20.133.126:41065/ using document root <none> and password file <none>
I20260927 13:55:15.562284 11339 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:15.562387 11339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:15.562717 11339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:15.565845 11339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/data/instance:
uuid: "93a5c043e76948c4ad30517a089d3543"
format_stamp: "Formatted at 2026-09-27 13:55:15 on dist-test-slave-q25r"
server_key: "9d7eacb40a9e7dbfa922f7d2ceb34113"
server_key_iv: "6768bcca4951d04a45a6b2b4d0047911"
server_key_version: "encryptionkey@0"
I20260927 13:55:15.566672 11339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/wal/instance:
uuid: "93a5c043e76948c4ad30517a089d3543"
format_stamp: "Formatted at 2026-09-27 13:55:15 on dist-test-slave-q25r"
server_key: "9d7eacb40a9e7dbfa922f7d2ceb34113"
server_key_iv: "6768bcca4951d04a45a6b2b4d0047911"
server_key_version: "encryptionkey@0"
I20260927 13:55:15.572715 11339 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 13:55:15.577102 11354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:15.578877 11339 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:55:15.579097 11339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/wal
uuid: "93a5c043e76948c4ad30517a089d3543"
format_stamp: "Formatted at 2026-09-27 13:55:15 on dist-test-slave-q25r"
server_key: "9d7eacb40a9e7dbfa922f7d2ceb34113"
server_key_iv: "6768bcca4951d04a45a6b2b4d0047911"
server_key_version: "encryptionkey@0"
I20260927 13:55:15.579329 11339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:15.596107 11339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:15.600844 11339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:15.601118 11339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:15.620990 11339 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.126:44561
I20260927 13:55:15.621037 11405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.126:44561 every 8 connection(s)
I20260927 13:55:15.622442 11339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/data/info.pb
I20260927 13:55:15.627586 11406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:15.627995 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 11339
I20260927 13:55:15.628170 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-0/wal/instance
I20260927 13:55:15.628649 21013 external_mini_cluster.cc:1428] Setting key b754869e20b457958308ddf8e4996b39
I20260927 13:55:15.635571 11406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.133.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.125@KRBTEST.COM" created.
Sep 27 13:55:15 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517315, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:55:15.664417 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:15.639215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:44561 (local address 127.0.0.1:42054)
0927 13:55:15.640135 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:15.640190 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:15.642677 (+ 2487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:15.643958 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:15.643964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:15.644716 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:15.645135 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:15.645154 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:15.647689 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:15.647694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:15.649058 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:15.649072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:15.650611 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:15.650616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:15.650806 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:15.651536 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:15.651567 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:15.655612 (+ 4045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:15.659394 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:15.659408 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:15.659424 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:15.659906 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:15.660365 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:15.660370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:15.660374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:15.660545 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:15.661455 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:15.661465 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:15.661700 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 13:55:15.661969 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 13:55:15.662727 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":207,"thread_start_us":380,"threads_started":1}
I20260927 13:55:15.664449 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:15.640687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.20.133.126:44561)
0927 13:55:15.641613 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:15.641624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:15.641699 (+ 75us) server_negotiation.cc:407] Connection header received
0927 13:55:15.643197 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:15.643225 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:15.643635 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:15.643901 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:15.645411 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:15.647463 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:15.649254 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:15.650322 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:15.650580 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:15.655920 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:15.655932 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:15.655949 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:15.655997 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:15.659206 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:15.660057 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:15.660065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:15.660068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:15.660239 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:15.660654 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:15.660657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:15.660660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:15.661308 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:15.661532 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:15.662138 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 13:55:15.662582 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":629,"thread_start_us":372,"threads_started":1}
I20260927 13:55:15.668640 11406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:55:15.670166 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:15.669669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51470)
0927 13:55:15.669880 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:15.670016 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 13:55:15.670544 11358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
I20260927 13:55:15.672387 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:15.671934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51476)
0927 13:55:15.672113 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:15.672192 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 13:55:15.672600 11406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:35957: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab.
Entry for principal kudu/127.20.133.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab.
I20260927 13:55:15.685578 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.125:35957
--webserver_interface=127.20.133.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab
--principal=kudu/127.20.133.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:44561,127.20.133.125:35957,127.20.133.124:46607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:55:15.722029 11406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } attempt: 1
I20260927 13:55:15.723420 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:15.722890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51484)
0927 13:55:15.723113 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:15.723262 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 13:55:15.724895 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:15.724499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51498)
0927 13:55:15.724657 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:15.724738 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 13:55:15.725116 11406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:35957: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
I20260927 13:55:15.793390 11406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } attempt: 2
I20260927 13:55:15.794955 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:15.794331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51510)
0927 13:55:15.794630 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:15.794751 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 13:55:15.796156 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:15.795772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51514)
0927 13:55:15.795922 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:15.796012 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 13:55:15.796416 11406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:35957: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
I20260927 13:55:15.887681 11406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } attempt: 3
I20260927 13:55:15.889189 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:15.888642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51516)
0927 13:55:15.888888 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:15.889015 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 13:55:15.890421 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:15.889994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51526)
0927 13:55:15.890176 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:15.890258 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 13:55:15.890645 11406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:35957: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
W20260927 13:55:16.019775 11413 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:55:16.020153 11413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:16.020236 11413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:16.020360 11413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:16.027930 11413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:16.028045 11413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:16.028113 11413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:16.028174 11413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:16.028242 11413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:16.033936 11406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } attempt: 4
I20260927 13:55:16.035454 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.034791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51542)
0927 13:55:16.035162 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.035282 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260927 13:55:16.036595 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.036208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51546)
0927 13:55:16.036371 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.036467 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 13:55:16.036826 11406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.125:35957: Network error: Client connection negotiation failed: client connection to 127.20.133.125:35957: connect: Connection refused (error 111)
I20260927 13:55:16.039335 11413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:44561,127.20.133.125:35957,127.20.133.124:46607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.125.keytab
--principal=kudu/127.20.133.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.125:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.133.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:16.040678 11413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:16.042006 11413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:16.054466 11419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:16.054561 11418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:16.054576 11421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:16.055639 11413 server_base.cc:1034] running on GCE node
I20260927 13:55:16.056564 11413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:16.058188 11413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:16.059484 11413 hybrid_clock.cc:648] HybridClock initialized: now 1790517316059382 us; error 53 us; skew 500 ppm
I20260927 13:55:16.059863 11413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517316, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:16.063340 11413 init.cc:377] Logged in from keytab as kudu/127.20.133.125@KRBTEST.COM (short username kudu)
I20260927 13:55:16.065124 11413 webserver.cc:460] Webserver started at http://127.20.133.125:39327/ using document root <none> and password file <none>
I20260927 13:55:16.065878 11413 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:16.065986 11413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:16.066296 11413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:16.069437 11413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/data/instance:
uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb"
format_stamp: "Formatted at 2026-09-27 13:55:16 on dist-test-slave-q25r"
server_key: "b2e348fb6a1ebdd2b104d27d2e0cbae6"
server_key_iv: "bb9f6188daeecc6af2c47ae7dd5acbba"
server_key_version: "encryptionkey@0"
I20260927 13:55:16.070245 11413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/wal/instance:
uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb"
format_stamp: "Formatted at 2026-09-27 13:55:16 on dist-test-slave-q25r"
server_key: "b2e348fb6a1ebdd2b104d27d2e0cbae6"
server_key_iv: "bb9f6188daeecc6af2c47ae7dd5acbba"
server_key_version: "encryptionkey@0"
I20260927 13:55:16.075989 11413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 13:55:16.080217 11428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:16.082044 11413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:55:16.082262 11413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/wal
uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb"
format_stamp: "Formatted at 2026-09-27 13:55:16 on dist-test-slave-q25r"
server_key: "b2e348fb6a1ebdd2b104d27d2e0cbae6"
server_key_iv: "bb9f6188daeecc6af2c47ae7dd5acbba"
server_key_version: "encryptionkey@0"
I20260927 13:55:16.082485 11413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:16.097818 11413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:16.102200 11413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:16.102480 11413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:16.121893 11413 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.125:35957
I20260927 13:55:16.122008 11479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.125:35957 every 8 connection(s)
I20260927 13:55:16.123298 11413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/data/info.pb
I20260927 13:55:16.128086 11480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:16.132522 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 11413
I20260927 13:55:16.132752 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-1/wal/instance
I20260927 13:55:16.133356 21013 external_mini_cluster.cc:1428] Setting key 98c962d1403497f89b2ef857042690cc
I20260927 13:55:16.137132 11480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517316, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:55:16.164224 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.141210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.20.133.126:44561)
0927 13:55:16.141482 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.141490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.142895 (+ 1405us) server_negotiation.cc:407] Connection header received
0927 13:55:16.145855 (+ 2960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.145862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.145967 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.146093 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.148475 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.149518 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.151412 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.151959 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.152107 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.156888 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.156895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.156899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.156948 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.159829 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.161171 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.161176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.161179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.161343 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.161938 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.161943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.161945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.162438 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.162573 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.163680 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.163936 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
WARNING: no policy specified for kudu/127.20.133.124@KRBTEST.COM; defaulting to no policy
I20260927 13:55:16.165561 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.141670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:44561 (local address 127.0.0.1:42068)
0927 13:55:16.142649 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.142710 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:16.145386 (+ 2676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:16.146435 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:16.146450 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:16.147325 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:16.148225 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.148248 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.149745 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.149750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.151217 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.151240 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.152279 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.152286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.152593 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.153403 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:16.153437 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:16.156491 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:16.160081 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.160100 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.160121 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.160961 (+ 840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.161482 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.161487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.161491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.161792 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.162584 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:16.162592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:16.163156 (+ 564us) client_negotiation.cc:769] Sending connection context
0927 13:55:16.163535 (+ 379us) client_negotiation.cc:240] Negotiation successful
0927 13:55:16.164146 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":371,"threads_started":1}
I20260927 13:55:16.167717 11480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.133.124@KRBTEST.COM" created.
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517316, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
I20260927 13:55:16.185745 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.168861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51556)
0927 13:55:16.169104 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.169126 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:16.169262 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:16.170499 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:16.170504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:16.171004 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:16.171218 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.171230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.172884 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.172890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.174078 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.174105 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.175052 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.175057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.175242 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.175945 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:16.175967 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:16.178774 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:16.182283 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.182288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.182293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.182874 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.183355 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.183360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.183363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.183553 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.184540 (+ 987us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:16.184547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:16.184715 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 13:55:16.184856 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 13:55:16.185215 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 13:55:16.185860 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.169132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.20.133.125:35957)
0927 13:55:16.169913 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.169921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.169956 (+ 35us) server_negotiation.cc:407] Connection header received
0927 13:55:16.170013 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.170020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.170292 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.170445 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.171437 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.172731 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.174245 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.174894 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.174993 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.179074 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.179102 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.179119 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.179158 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.182078 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.183041 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.183050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.183053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.183242 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.183730 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.183735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.183738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.184263 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.184632 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.185027 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.185456 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":589,"thread_start_us":430,"threads_started":1}
I20260927 13:55:16.189258 11480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:55:16.190560 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.190053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52856)
0927 13:55:16.190249 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.190386 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 13:55:16.190948 11429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
I20260927 13:55:16.193114 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.192667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52866)
0927 13:55:16.192875 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.192965 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 13:55:16.193364 11480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46607: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.133.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab.
Entry for principal kudu/127.20.133.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab.
I20260927 13:55:16.207795 21013 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
/tmp/dist-test-taskM7FxcK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.124:46607
--webserver_interface=127.20.133.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab
--principal=kudu/127.20.133.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.133.84:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.133.126:44561,127.20.133.125:35957,127.20.133.124:46607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:55:16.242707 11480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } attempt: 1
I20260927 13:55:16.244544 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.243721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52870)
0927 13:55:16.244099 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.244280 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 13:55:16.247611 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.245786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52876)
0927 13:55:16.247323 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.247445 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1390}
W20260927 13:55:16.250129 11480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46607: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
I20260927 13:55:16.318444 11480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } attempt: 2
I20260927 13:55:16.320000 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.319328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52886)
0927 13:55:16.319600 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.319747 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 13:55:16.321296 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.320823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52896)
0927 13:55:16.320985 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.321130 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 13:55:16.321578 11480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46607: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
I20260927 13:55:16.336119 11406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } attempt: 5
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517315, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
I20260927 13:55:16.352550 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.337268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.20.133.125:35957)
0927 13:55:16.337486 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.337491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.337506 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:55:16.337576 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.337580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.337655 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.337733 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.339138 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.339877 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.341848 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.342306 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.342410 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.346548 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.346553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.346557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.346598 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.349517 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.350650 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.350654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.350657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.350782 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.351214 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.351218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.351220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.351550 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.351666 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.352078 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.352258 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 13:55:16.352747 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.336855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51572)
0927 13:55:16.337060 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.337079 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:16.337188 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:16.337883 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:16.337890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:16.338506 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:16.338841 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.338859 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.340049 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.340056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.341658 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.341677 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.342478 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.342484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.342725 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.343490 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:16.343514 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:16.346288 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:16.349687 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.349692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.349697 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.350457 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.350903 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.350908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.350911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.351103 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.351662 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:16.351666 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:16.351845 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 13:55:16.352001 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 13:55:16.352389 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 13:55:16.353845 11406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:55:16.354936 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.354490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52904)
0927 13:55:16.354684 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.354781 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 13:55:16.355818 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.355473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52920)
0927 13:55:16.355600 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.355698 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 13:55:16.356046 11406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46607: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
I20260927 13:55:16.407313 11406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } attempt: 1
I20260927 13:55:16.408643 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.408021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52934)
0927 13:55:16.408240 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.408393 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 13:55:16.409667 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.409293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52946)
0927 13:55:16.409436 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.409535 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 13:55:16.409883 11406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46607: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
I20260927 13:55:16.412807 11480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } attempt: 3
I20260927 13:55:16.413944 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.413408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52950)
0927 13:55:16.413634 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.413754 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 13:55:16.415107 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.414638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52958)
0927 13:55:16.414883 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.414973 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260927 13:55:16.415349 11480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46607: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
I20260927 13:55:16.478106 11406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } attempt: 2
I20260927 13:55:16.479566 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.478960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52974)
0927 13:55:16.479220 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.479344 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 13:55:16.480723 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.480269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52978)
0927 13:55:16.480467 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.480580 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 13:55:16.481001 11406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46607: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
I20260927 13:55:16.558588 11480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } attempt: 4
I20260927 13:55:16.559934 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.559327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52986)
0927 13:55:16.559587 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.559721 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 13:55:16.561121 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.560701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52994)
0927 13:55:16.560863 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.560974 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 13:55:16.561357 11480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46607: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
W20260927 13:55:16.577288 11487 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:55:16.577685 11487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:55:16.577769 11487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:55:16.577865 11487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:55:16.586728 11487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:55:16.586840 11487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:55:16.586901 11487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:55:16.586952 11487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:55:16.587014 11487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:55:16.587227 11406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } attempt: 3
I20260927 13:55:16.588896 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.588062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:52996)
0927 13:55:16.588525 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.588673 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 13:55:16.590143 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.589698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:53000)
0927 13:55:16.589901 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.590004 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 13:55:16.590389 11406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.133.124:46607: Network error: Client connection negotiation failed: client connection to 127.20.133.124:46607: connect: Connection refused (error 111)
I20260927 13:55:16.598675 11487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.133.84:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.133.126:44561,127.20.133.125:35957,127.20.133.124:46607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/krb5kdc/kudu_127.20.133.124.keytab
--principal=kudu/127.20.133.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.133.124:46607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.133.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:55:16.600072 11487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:55:16.601542 11487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:55:16.613451 11493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:16.613464 11495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:55:16.613504 11492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:55:16.614567 11487 server_base.cc:1034] running on GCE node
I20260927 13:55:16.615371 11487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:55:16.616945 11487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:55:16.618196 11487 hybrid_clock.cc:648] HybridClock initialized: now 1790517316618100 us; error 72 us; skew 500 ppm
I20260927 13:55:16.618544 11487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517316, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:55:16.622062 11487 init.cc:377] Logged in from keytab as kudu/127.20.133.124@KRBTEST.COM (short username kudu)
I20260927 13:55:16.623852 11487 webserver.cc:460] Webserver started at http://127.20.133.124:39771/ using document root <none> and password file <none>
I20260927 13:55:16.624629 11487 fs_manager.cc:359] Metadata directory not provided
I20260927 13:55:16.624743 11487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:55:16.625053 11487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:55:16.627978 11487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/data/instance:
uuid: "280763518a004088b79eccdfcd5d67b4"
format_stamp: "Formatted at 2026-09-27 13:55:16 on dist-test-slave-q25r"
server_key: "bc8178dbe60300e68242ccca53d82b25"
server_key_iv: "91eb7ddf6a9810be936c48739c0bdcf2"
server_key_version: "encryptionkey@0"
I20260927 13:55:16.628765 11487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/wal/instance:
uuid: "280763518a004088b79eccdfcd5d67b4"
format_stamp: "Formatted at 2026-09-27 13:55:16 on dist-test-slave-q25r"
server_key: "bc8178dbe60300e68242ccca53d82b25"
server_key_iv: "91eb7ddf6a9810be936c48739c0bdcf2"
server_key_version: "encryptionkey@0"
I20260927 13:55:16.633776 11487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 13:55:16.637274 11502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:55:16.638633 11487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:55:16.638798 11487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/data,/tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/wal
uuid: "280763518a004088b79eccdfcd5d67b4"
format_stamp: "Formatted at 2026-09-27 13:55:16 on dist-test-slave-q25r"
server_key: "bc8178dbe60300e68242ccca53d82b25"
server_key_iv: "91eb7ddf6a9810be936c48739c0bdcf2"
server_key_version: "encryptionkey@0"
I20260927 13:55:16.639010 11487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:55:16.655544 11487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:55:16.659704 11487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:55:16.659960 11487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:55:16.677273 11487 rpc_server.cc:307] RPC server started. Bound to: 127.20.133.124:46607
I20260927 13:55:16.677292 11553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.133.124:46607 every 8 connection(s)
I20260927 13:55:16.678642 11487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/data/info.pb
I20260927 13:55:16.682220 11554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:55:16.688022 21013 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu as pid 11487
I20260927 13:55:16.688222 21013 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM7FxcK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516948056776-21013-0/minicluster-data/master-2/wal/instance
I20260927 13:55:16.688822 21013 external_mini_cluster.cc:1428] Setting key 96ab52f1cc292acca868e6e079f2010f
I20260927 13:55:16.689287 11554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517316, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
I20260927 13:55:16.708240 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.692084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.20.133.126:44561)
0927 13:55:16.692913 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.692919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.693378 (+ 459us) server_negotiation.cc:407] Connection header received
0927 13:55:16.694084 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.694089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.694175 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.694269 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.695586 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.696416 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.697872 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.698278 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.698385 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.702113 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.702119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.702123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.702162 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.705000 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.705878 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.705883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.705885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.706019 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.706441 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.706446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.706448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.706742 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.706880 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.707589 (+ 709us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.707846 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":604,"threads_started":1}
I20260927 13:55:16.709321 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.692040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.126:44561 (local address 127.0.0.1:42084)
0927 13:55:16.693232 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.693271 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:16.693818 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:16.694360 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:16.694367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:16.695049 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:16.695417 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.695434 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.696566 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.696570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.697738 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.697749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.698511 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.698519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.698706 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.699408 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:16.699438 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:16.701878 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:16.705149 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.705172 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.705186 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.705756 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.706125 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.706130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.706134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.706334 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.706865 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:16.706874 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:16.707246 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 13:55:16.707472 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 13:55:16.708019 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":376,"threads_started":1}
I20260927 13:55:16.709339 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.690685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.20.133.124:46607)
0927 13:55:16.691458 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.691470 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.691507 (+ 37us) server_negotiation.cc:407] Connection header received
0927 13:55:16.693774 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.693795 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.694039 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.694218 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.695757 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.696943 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.698177 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.698708 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.698823 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.702639 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.702650 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.702670 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.702707 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.705614 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.706337 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.706344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.706347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.706486 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.706845 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.706849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.706851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.707303 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.707604 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.707939 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.708223 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"spinlock_wait_cycles":18688,"thread_start_us":385,"threads_started":1}
I20260927 13:55:16.710865 11554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517316, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.126@KRBTEST.COM
I20260927 13:55:16.723708 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.711676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.125:35957 (local address 127.0.0.1:51574)
0927 13:55:16.711832 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.711850 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:16.711934 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:16.712245 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:16.712250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:16.712716 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:16.712911 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.712922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.713929 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.713935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.714938 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.714950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.715598 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.715604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.715809 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.716379 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:16.716402 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:16.718572 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:16.721525 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.721530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.721534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.721951 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.722311 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.722314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.722316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.722467 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.723020 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:16.723023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:16.723147 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 13:55:16.723264 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 13:55:16.723456 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 13:55:16.723942 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.711675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.20.133.125:35957)
0927 13:55:16.711860 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.711866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.711886 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:55:16.712040 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.712046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.712123 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.712202 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.713038 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.713798 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.715062 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.715457 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.715540 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.718743 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.718748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.718751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.718781 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.721398 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.722053 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.722058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.722061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.722214 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.722571 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.722576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.722580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.722917 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.723058 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.723370 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.723577 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:55:16.724817 11554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:55:16.726914 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.712706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.20.133.126:44561)
0927 13:55:16.712845 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.712849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.712863 (+ 14us) server_negotiation.cc:407] Connection header received
0927 13:55:16.713008 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.713013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.713100 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.713197 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.714313 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.714935 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.716381 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.716801 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.716889 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.720939 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.720946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.720950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.720987 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.724362 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.725048 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.725053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.725057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.725181 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.725564 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.725570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.725573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.725872 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.726122 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.726422 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.726637 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517316, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.124@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.133.125@KRBTEST.COM
I20260927 13:55:16.737447 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.725394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:53030)
0927 13:55:16.725533 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.725550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:16.725630 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:16.725932 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:16.725937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:16.726288 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:16.726419 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.726427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.727443 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.727448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.728481 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.728492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.729107 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.729112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.729314 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.729805 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:16.729821 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:16.731793 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:16.734616 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.734622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.734626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.735099 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.735520 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.735524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.735526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.735658 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.736226 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:16.736231 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:16.736422 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 13:55:16.736627 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 13:55:16.737128 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":199}
I20260927 13:55:16.737483 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.725569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53030 (local address 127.20.133.124:46607)
0927 13:55:16.725733 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.725739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.725753 (+ 14us) server_negotiation.cc:407] Connection header received
0927 13:55:16.725787 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.725790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.725836 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.725906 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.726534 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.727336 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.728592 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.728985 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.729093 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.731999 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.732005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.732008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.732037 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.734461 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.735231 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.735237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.735240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.735401 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.735758 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.735777 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.735781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.736104 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.736218 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.736714 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.737027 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260927 13:55:16.742977 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.728383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.20.133.125:35957)
0927 13:55:16.728532 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.728538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.728554 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:55:16.728699 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.728704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.728757 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.728830 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.729942 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.730627 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.732010 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.732443 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.732541 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.736913 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.736924 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.736928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.737053 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.740124 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.741016 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.741020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.741022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.741162 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.741602 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.741605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.741607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.741876 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.742118 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.742495 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.742673 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:55:16.743300 11554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4: Bootstrap starting.
I20260927 13:55:16.747205 11554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:16.747924 21013 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:55:16.748448 11554 log.cc:824] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:16.751504 11554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4: No bootstrap required, opened a new log
I20260927 13:55:16.750794 11381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "93a5c043e76948c4ad30517a089d3543"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42098
I20260927 13:55:16.762338 11554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } }
I20260927 13:55:16.762794 11554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280763518a004088b79eccdfcd5d67b4, State: Initialized, Role: FOLLOWER
I20260927 13:55:16.763320 11554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } }
I20260927 13:55:16.764318 11561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } } }
I20260927 13:55:16.764683 11561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:16.765647 11554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:16.767586 11406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } attempt: 4
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517315, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.126@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
I20260927 13:55:16.783641 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.768589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.20.133.124:46607)
0927 13:55:16.768747 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.768753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.768773 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:55:16.768831 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.768836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.768914 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.769012 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.769790 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.770870 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.772045 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.772669 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.772786 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.776682 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.776691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.776696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.776734 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.780493 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.781214 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.781224 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.781228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.781409 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.781844 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.781849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.781853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.782379 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.782569 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.782961 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.783189 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:55:16.783660 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.768313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:53042)
0927 13:55:16.768531 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.768552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:16.768690 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:16.769042 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:16.769046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:16.769467 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:16.769634 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.769642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.771012 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.771018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.771913 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.771925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.772814 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.772820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.772996 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.773666 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:16.773702 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:16.776479 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:16.780659 (+ 4180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.780665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.780670 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.781075 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.781518 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.781540 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.781544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.781721 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.782551 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:16.782556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:16.782730 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 13:55:16.782864 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 13:55:16.783178 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 13:55:16.788522 11406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543: Bootstrap starting.
I20260927 13:55:16.776794 11487 master_runner.cc:186] Error getting master registration for 127.20.133.126:44561: OK, instance_id { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" instance_seqno: 1790517315615310 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:55:16.790253 11487 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 13:55:16.792368 11406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543: Neither blocks nor log segments found. Creating new log.
W20260927 13:55:16.792953 11572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:55:16.793076 11572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:55:16.793568 11406 log.cc:824] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:16.796504 11406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543: No bootstrap required, opened a new log
I20260927 13:55:16.801723 11406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } }
I20260927 13:55:16.802026 11406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a5c043e76948c4ad30517a089d3543, State: Initialized, Role: FOLLOWER
I20260927 13:55:16.802414 11406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } }
I20260927 13:55:16.803303 11575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } } }
I20260927 13:55:16.803696 11575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:16.804435 11406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:16.804606 11381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 13:55:16.804844 11381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:16.808434 11381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } }
I20260927 13:55:16.810374 11381 leader_election.cc:288] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [CANDIDATE]: Term 1 election: Requested vote from peers bc4c03dfd2c34ed79af70c2bdea42dfb (127.20.133.125:35957), 280763518a004088b79eccdfcd5d67b4 (127.20.133.124:46607)
I20260927 13:55:16.811865 11529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93a5c043e76948c4ad30517a089d3543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "280763518a004088b79eccdfcd5d67b4"
I20260927 13:55:16.812433 11529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:16.815878 11529 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a5c043e76948c4ad30517a089d3543 in term 1.
I20260927 13:55:16.817826 11355 leader_election.cc:302] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 280763518a004088b79eccdfcd5d67b4, 93a5c043e76948c4ad30517a089d3543; no voters:
I20260927 13:55:16.818323 11575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:55:16.815492 11339 master_runner.cc:186] Error getting master registration for 127.20.133.125:35957: OK, instance_id { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" instance_seqno: 1790517316116592 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:55:16.826359 11339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 13:55:16.826722 11586 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:55:16.826853 11586 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:55:16.828748 11575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [term 1 LEADER]: Becoming Leader. State: Replica: 93a5c043e76948c4ad30517a089d3543, State: Running, Role: LEADER
I20260927 13:55:16.829875 11575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } }
I20260927 13:55:16.845073 11589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93a5c043e76948c4ad30517a089d3543. Latest consensus state: current_term: 1 leader_uuid: "93a5c043e76948c4ad30517a089d3543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } } }
I20260927 13:55:16.845709 11589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:16.848836 11596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:55:16.851090 11596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 13:55:16.854423 11360 tablet.cc:2369] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:55:16.860581 11480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } attempt: 5
I20260927 13:55:16.873440 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.835334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.20.133.126:44561)
0927 13:55:16.835518 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.835523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.847781 (+ 12258us) server_negotiation.cc:407] Connection header received
0927 13:55:16.855134 (+ 7353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.855140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.855231 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.855303 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.857505 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.858612 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.860450 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.861094 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.861200 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.865357 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.865364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.865369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.865435 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.869474 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.870368 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.870374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.870378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.870534 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.871275 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.871280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.871283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.871668 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.872173 (+ 505us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.872762 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.873005 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:55:16.879316 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.852447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.20.133.124:46607)
0927 13:55:16.852637 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.852643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.853804 (+ 1161us) server_negotiation.cc:407] Connection header received
0927 13:55:16.855199 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.855205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.855299 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.855362 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.858739 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.859893 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.863066 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.863783 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.863892 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.869976 (+ 6084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.869984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.869989 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.870045 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.873476 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.875859 (+ 2383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.875868 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.875873 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.876120 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.877152 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.877158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.877171 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.877540 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.877813 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.878668 (+ 855us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.878924 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:55:16.882663 11529 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [term 1 FOLLOWER]: Refusing update from remote peer 93a5c043e76948c4ad30517a089d3543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:16.884368 11589 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 27 13:55:16 dist-test-slave-q25r krb5kdc[11326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790517316, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.133.125@KRBTEST.COM for kudu/127.20.133.124@KRBTEST.COM
I20260927 13:55:16.894811 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.834904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.20.133.125:35957)
0927 13:55:16.835109 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.835116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.851724 (+ 16608us) server_negotiation.cc:407] Connection header received
0927 13:55:16.853457 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.853464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.853554 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.853629 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.860506 (+ 6877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.861540 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.871735 (+ 10195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.872464 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.872570 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.883998 (+ 11428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.884006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.884011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.884052 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.887798 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.889287 (+ 1489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.889293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.889296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.889468 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.892643 (+ 3175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.892649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.892652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.893035 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.893274 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.893945 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.894189 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:55:16.903672 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.861240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.133.124:46607 (local address 127.0.0.1:53062)
0927 13:55:16.861641 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 13:55:16.861661 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:55:16.861750 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:55:16.882137 (+ 20387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:55:16.882145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:55:16.882992 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:55:16.883269 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.883283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.885077 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.885087 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.886602 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:55:16.886620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.887757 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.887763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:55:16.888036 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.888611 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:55:16.888634 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:55:16.891303 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:55:16.897959 (+ 6656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.897966 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.897970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.898964 (+ 994us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.899950 (+ 986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:55:16.899956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:55:16.899959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:55:16.900283 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:55:16.901693 (+ 1410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:55:16.901699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:55:16.902025 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 13:55:16.902269 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 13:55:16.902852 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260927 13:55:16.906191 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:16.861767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.20.133.124:46607)
0927 13:55:16.880072 (+ 18305us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:16.880105 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:16.880124 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:55:16.880214 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:16.880243 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:16.881930 (+ 1687us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:16.882114 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:16.883537 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.884879 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.886866 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:16.887580 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:16.888355 (+ 775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:16.891573 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:55:16.891591 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:55:16.891596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:55:16.891638 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:55:16.897678 (+ 6040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.899322 (+ 1644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.899328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.899331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.899726 (+ 395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:55:16.900597 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:55:16.900603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:55:16.900606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:55:16.901434 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:55:16.904387 (+ 2953us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:16.904737 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 13:55:16.905355 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18151,"thread_start_us":21966,"threads_started":1}
W20260927 13:55:16.912146 11508 tablet.cc:2369] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:55:16.912613 11480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb: Bootstrap starting.
I20260927 13:55:16.920311 11480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb: Neither blocks nor log segments found. Creating new log.
I20260927 13:55:16.922710 11480 log.cc:824] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb: Log is configured to *not* fsync() on all Append() calls
I20260927 13:55:16.928217 11561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93a5c043e76948c4ad30517a089d3543. Latest consensus state: current_term: 1 leader_uuid: "93a5c043e76948c4ad30517a089d3543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } } }
I20260927 13:55:16.928557 11561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:16.933586 11599 mvcc.cc:205] Tried to move back new op lower bound from 7333958929938743296 to 7333958929744809984. Current Snapshot: MvccSnapshot[applied={T|T < 7333958929938743296}]
I20260927 13:55:16.936221 11596 catalog_manager.cc:1347] Generated new cluster ID: a15c5f6d452a44a9bbde24e7b0ed9f93
I20260927 13:55:16.936326 11596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:55:16.937227 11589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93a5c043e76948c4ad30517a089d3543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } } }
I20260927 13:55:16.937503 11589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:16.943701 11480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb: No bootstrap required, opened a new log
I20260927 13:55:16.957386 11561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93a5c043e76948c4ad30517a089d3543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } } }
I20260927 13:55:16.957695 11561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:16.955412 11480 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } }
I20260927 13:55:16.959262 11480 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4c03dfd2c34ed79af70c2bdea42dfb, State: Initialized, Role: FOLLOWER
I20260927 13:55:16.959805 11480 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } }
I20260927 13:55:16.964465 11596 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:55:16.966435 11596 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:55:16.969352 11605 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } } }
I20260927 13:55:16.969938 11605 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:16.970577 11480 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:55:16.971014 11454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:55:16.975005 11454 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [term 1 FOLLOWER]: Refusing update from remote peer 93a5c043e76948c4ad30517a089d3543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:55:16.975718 11455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93a5c043e76948c4ad30517a089d3543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb"
I20260927 13:55:16.976115 11589 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:55:16.979463 11455 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a5c043e76948c4ad30517a089d3543 in term 1.
I20260927 13:55:16.994896 11596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543: Generated new TSK 0
I20260927 13:55:16.996115 11596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:55:17.046196 11605 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93a5c043e76948c4ad30517a089d3543. Latest consensus state: current_term: 1 leader_uuid: "93a5c043e76948c4ad30517a089d3543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } } }
I20260927 13:55:17.046164 11616 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93a5c043e76948c4ad30517a089d3543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } } }
I20260927 13:55:17.046653 11605 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:17.046684 11616 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:55:17.048156 11575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93a5c043e76948c4ad30517a089d3543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a5c043e76948c4ad30517a089d3543" member_type: VOTER last_known_addr { host: "127.20.133.126" port: 44561 } } peers { permanent_uuid: "bc4c03dfd2c34ed79af70c2bdea42dfb" member_type: VOTER last_known_addr { host: "127.20.133.125" port: 35957 } } peers { permanent_uuid: "280763518a004088b79eccdfcd5d67b4" member_type: VOTER last_known_addr { host: "127.20.133.124" port: 46607 } } }
I20260927 13:55:17.048588 11575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a5c043e76948c4ad30517a089d3543 [sys.catalog]: This master's current role is: LEADER
I20260927 13:55:17.068759 11622 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260927 13:55:17.115340 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.102492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.20.133.125:35957)
0927 13:55:17.102977 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.102985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.113389 (+ 10404us) server_negotiation.cc:407] Connection header received
0927 13:55:17.113696 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.113703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.113795 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.113873 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.115090 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51610: BlockingRecv error: recv got EOF from 127.0.0.1:51610 (error 108)
Metrics: {"server-negotiator.queue_time_us":299}
W20260927 13:55:17.118001 11557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.113540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.20.133.124:46607)
0927 13:55:17.113711 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.113718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.116303 (+ 2585us) server_negotiation.cc:407] Connection header received
0927 13:55:17.116578 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.116584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.116667 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.116739 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.117737 (+ 998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53068: BlockingRecv error: recv got EOF from 127.0.0.1:53068 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 13:55:17.120045 11559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.103138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.20.133.126:44561)
0927 13:55:17.103341 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.103347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.113142 (+ 9795us) server_negotiation.cc:407] Connection header received
0927 13:55:17.118645 (+ 5503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.118653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.118768 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.118876 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.119802 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42122: BlockingRecv error: recv got EOF from 127.0.0.1:42122 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260927 13:55:17.142735 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.123540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.20.133.125:35957)
0927 13:55:17.123735 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.123741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.140941 (+ 17200us) server_negotiation.cc:407] Connection header received
0927 13:55:17.141164 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.141170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.141308 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.141400 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.142453 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51622: BlockingRecv error: recv got EOF from 127.0.0.1:51622 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 13:55:17.160213 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.158326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.20.133.125:35957)
0927 13:55:17.158516 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.158523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.158976 (+ 453us) server_negotiation.cc:407] Connection header received
0927 13:55:17.159164 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.159169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.159245 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.159324 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.160010 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51628: BlockingRecv error: recv got EOF from 127.0.0.1:51628 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 13:55:17.167506 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.165441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.20.133.125:35957)
0927 13:55:17.165631 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.165638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.166096 (+ 458us) server_negotiation.cc:407] Connection header received
0927 13:55:17.166300 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.166306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.166408 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.166496 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.167248 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51632: BlockingRecv error: recv got EOF from 127.0.0.1:51632 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 13:55:17.180152 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.178175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.20.133.125:35957)
0927 13:55:17.178416 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.178423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.178845 (+ 422us) server_negotiation.cc:407] Connection header received
0927 13:55:17.179027 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.179032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.179109 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.179190 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.179900 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51642: BlockingRecv error: recv got EOF from 127.0.0.1:51642 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260927 13:55:17.193274 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.191220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.20.133.125:35957)
0927 13:55:17.191425 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.191432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.191923 (+ 491us) server_negotiation.cc:407] Connection header received
0927 13:55:17.192120 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.192124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.192204 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.192286 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.193030 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51650: BlockingRecv error: recv got EOF from 127.0.0.1:51650 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 13:55:17.214847 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.212628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.20.133.125:35957)
0927 13:55:17.212850 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.212859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.213373 (+ 514us) server_negotiation.cc:407] Connection header received
0927 13:55:17.213611 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.213616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.213709 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.213812 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.214605 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51666: BlockingRecv error: recv got EOF from 127.0.0.1:51666 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 13:55:17.252454 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.250337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.20.133.125:35957)
0927 13:55:17.250539 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.250547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.251061 (+ 514us) server_negotiation.cc:407] Connection header received
0927 13:55:17.251259 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.251265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.251342 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.251424 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.252137 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51672: BlockingRecv error: recv got EOF from 127.0.0.1:51672 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 13:55:17.328429 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.326126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.20.133.125:35957)
0927 13:55:17.326316 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.326321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.326864 (+ 543us) server_negotiation.cc:407] Connection header received
0927 13:55:17.327069 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.327074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.327188 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.327291 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.328183 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51686: BlockingRecv error: recv got EOF from 127.0.0.1:51686 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 13:55:17.461436 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.459611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.20.133.125:35957)
0927 13:55:17.459783 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.459787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.460161 (+ 374us) server_negotiation.cc:407] Connection header received
0927 13:55:17.460295 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.460299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.460427 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.460523 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.461235 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51690: BlockingRecv error: recv got EOF from 127.0.0.1:51690 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 13:55:17.722464 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:55:17.720832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.20.133.125:35957)
0927 13:55:17.721048 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:17.721055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:17.721154 (+ 99us) server_negotiation.cc:407] Connection header received
0927 13:55:17.721363 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:17.721368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:17.721442 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:17.721526 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:55:17.722252 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51704: BlockingRecv error: recv got EOF from 127.0.0.1:51704 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 13:55:17.795197 11572 catalog_manager.cc:1259] Loaded cluster ID: a15c5f6d452a44a9bbde24e7b0ed9f93
I20260927 13:55:17.795300 11572 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4: loading cluster ID for follower catalog manager: success
I20260927 13:55:17.797899 11572 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4: acquiring CA information for follower catalog manager: success
I20260927 13:55:17.799703 11572 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 280763518a004088b79eccdfcd5d67b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:55:18.073464 11622 catalog_manager.cc:1259] Loaded cluster ID: a15c5f6d452a44a9bbde24e7b0ed9f93
I20260927 13:55:18.073604 11622 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb: loading cluster ID for follower catalog manager: success
I20260927 13:55:18.076372 11622 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb: acquiring CA information for follower catalog manager: success
I20260927 13:55:18.078349 11622 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc4c03dfd2c34ed79af70c2bdea42dfb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:55:18.253151 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:18.242590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.20.133.125:35957)
0927 13:55:18.246239 (+ 3649us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:18.246250 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:18.246277 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:55:18.246360 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:18.246365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:18.246606 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:18.246768 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:18.247366 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:18.248666 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:18.250724 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:18.251203 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:18.251302 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:18.251868 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:18.252504 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:18.252628 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:18.252750 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 13:55:18.252824 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3482,"thread_start_us":3264,"threads_started":1}
I20260927 13:55:18.266613 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:18.259777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.20.133.124:46607)
0927 13:55:18.260398 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:18.260403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:18.260420 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:55:18.260513 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:18.260517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:18.260672 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:18.260814 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:18.261261 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:18.262006 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:18.264017 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:18.264507 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:18.264617 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:18.265269 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:18.265914 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:18.266042 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:18.266195 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 13:55:18.266279 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":396,"threads_started":1}
I20260927 13:55:18.288645 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:18.273913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.20.133.124:46607)
0927 13:55:18.274106 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:18.274112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:18.281496 (+ 7384us) server_negotiation.cc:407] Connection header received
0927 13:55:18.281798 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:18.281804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:18.281907 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:18.281968 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:18.282723 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:18.283869 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:18.286372 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:18.286991 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:18.287119 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:18.287775 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:18.287919 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:18.287982 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:18.288213 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 13:55:18.288315 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:55:18.289927 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:18.272861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.20.133.126:44561)
0927 13:55:18.281020 (+ 8159us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:18.281027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:18.281056 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:55:18.281194 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:18.281200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:18.281434 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:18.282541 (+ 1107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:18.282745 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:18.283854 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:18.286370 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:18.287017 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:18.287740 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:18.287934 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:18.288426 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:18.289443 (+ 1017us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:18.289519 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 13:55:18.289628 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8021,"thread_start_us":7903,"threads_started":1}
I20260927 13:55:18.295400 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:55:18.273109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.20.133.125:35957)
0927 13:55:18.273295 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 13:55:18.273301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:55:18.289095 (+ 15794us) server_negotiation.cc:407] Connection header received
0927 13:55:18.289329 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:55:18.289335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:55:18.289525 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:55:18.289629 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:55:18.290274 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:18.291246 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:18.292909 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:55:18.293628 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:55:18.293792 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:55:18.294479 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:55:18.294635 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:55:18.294746 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 13:55:18.294911 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 13:55:18.295013 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:55:18.295629 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 11339
I20260927 13:55:18.296710 11401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:18.421903 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 11339
I20260927 13:55:18.448496 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 11413
I20260927 13:55:18.449640 11475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:18.565954 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 11413
I20260927 13:55:18.582300 21013 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu pid 11487
I20260927 13:55:18.583408 11549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:55:18.690460 21013 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM7FxcK/build/asan/bin/kudu with pid 11487
2026-09-27T13:55:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4859 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9856 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (370585 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST